不多说,直接查询相关的内容
https://mvnrepository.com/search?q=spark
我们可以看到很多常见的东西,spark-core,spark-sql,spark-streaming等等
点开这些链接进入到版本列表界面,选择自己需要的配置
spark-core
spark-streaming
这里就不一一说明了,下面我直接列出一个已经测试成功的pom.xml
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.11.12</scala.version>
<spark.version>2.3.0</spark.version>
<hadoop.version>2.6.5</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
这里不建议在dependency中写死version,而是采用一下方式
<hadoop.version>2.6.5</hadoop.version>
在我的配置中有4个hadoop.version,如果写死了,那么后边要进行统一的hadoop升级,你该如何操作?一个一个找,然后修改4个位置?如果这里有上百个依赖,你该如何操作。
这个pom.xml修改完毕,保存后,我的eclipse上爆出了170个problems,但是不要慌,进行如下操作