版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
Spark 1+ 和 Spark 2+ 初始化SparkContext有所区别,现列出Spark 1.5.1 和 Spark 2+中初始化sc方式:
1. Spark 2+
先创建一个SparkSession对象:
通过config方法配置自定义设置,可以写多个config设置
val spark = SparkSession.builder() // Creates a [[SparkSession.Builder]] for constructing a [[SparkSession]].
.appName("Word Count") // Sets a name for the application, which will be shown in the Spark web UI. If no application name is set, a randomly generated name will be used.
.config("spark.some.config.option", "some-value") //Sets a config option. Options set using this method are automatically propagated to both `SparkConf` and SparkSession's own configuration.
.enableHiveSupport() // 启用HiveContext
.getOrCreate() //Gets an existing [[SparkSession]] or, if there is no existing one, creates a new one based on the options set in this builder.
val sc = spark.sparkContext
spark.sql("select 123") # 调用sql