scala 2.11.12 下载:https://www.scala-lang.org/download/
scala 2.11.12(Linux):scala-2.11.12.tgz
scala 2.11.12(windows):scala-2.11.12.zip
IDEA 新建一个 Maven项目
Maven项目创建成功提示
[INFO] BUILD SUCCESS
pom.xml 参考:
https://blog.csdn.net/qq262593421/article/details/105769886
创建Object对象
package com.xtd.spark
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext
/**
* sparkSQL example
*/
object SparkSQLAPP {
def main(args: Array[String]): Unit = {
// -Dspark.master=local
// D:/Hadoop/Spark/spark-2.4.0-bin-without-hadoop/examples/src/main/resources/employees.json
val path = args(0)
//val path = "file:///home/spark/examples/employees.json"
// 1、初始化配置
val sparkConf = new SparkConf().setAppName("SparkSQLAPP").setMaster("local[2]")
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
// 2、数据处理
val people = sqlContext.read.format("json").load(path)
people.printSchema()
people.show()
// 3、关闭资源
sc.stop()
}
}
点击右上角的对象名称,编辑项目配置,添加传递参数(本地文件添加前缀file:///)
file:///D:/Hadoop/Spark/spark-2.4.0-bin-without-hadoop/examples/src/main/resources/employees.json
people.json 文件 ,这个文件在spark安装文件根目录下的examples下可找到
{"name":"Michael"}
{"name":"Andy", "age":30}
{"name":"Justin", "age":19}
运行会出现以下成功提示