Maven调试Spark时遇到的问题
Pom.xml添加的内容
这个步骤是必须的,一定要确保这一步不然maven是不支持Scala语言开发的。
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
</dependency>
</dependencies>
<build>
<finalName>WordCount</finalName>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
更换maven仓库源
主要是更换maven的源,右键Maven项目找到Maven选项点击其子页的Open “settings.xml”或Create “settings.xml”,然后全选更换配置如下即可。
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<mirrors>
<!-- mirror
| Specifies a repository mirror site to use instead of a given repository. The repository that
| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
|
<mirror>
<id>mirrorId</id>
<mirrorOf>repositoryId</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://my.repository.com/repo/path</url>
</mirror>
-->
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>uk</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://uk.maven.org/maven2/</url>
</mirror>
<mirror>
<id>CN</id>
<name>OSChina Central</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>nexus</id>
<name>internal nexus repository</name>
<!-- <url>http://192.168.1.100:8081/nexus/content/groups/public/</url>-->
<url>http://repo.maven.apache.org/maven2</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
</settings>
设置Scala默认类型
Failed to locate the winutils binary
在调试程序的时候,出现Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries。通过百度查询发现程序需要根据HADOOP_HOME找到winutils.exe,我的Windows并没有配置该环境变量,所以程序报null\bin\winutils.exe。
经过上网查询找到了解决方案,过程如下:
- 下载winutils的windows版本
GitHub上,有人提供了winutils的windows的版本,链接地址是:文件下载地址,直接下载此项目的压缩包,下载后是文件名是hadoop-common-2.2.0-bin-master.zip,下载后解压即可。 - 配置环境变量
增加用户变量HADOOP_HOME,值是下载的压缩包解压后的目录。然后在系统变量path里增加%HADOOP_HOME%\bin 即可。 - 重启开发工具IDEA,再次运行程序,正常执行。