搭建Spark服务器环境

一、spark服务器环境:
1.配置java环境;
2.下载spark包和对应版本的hadoop包;
3.进入spark安装目录,在conf目录下spark-env.sh文件的最后加入环境变量:

LD_LIBRARY_PATH=$HADOOP_HOME/lib/native


4.执行sbin/start-master.sh和sbin/start-slave.sh spark://hostname:7077

启动spark并检查是否有错误和警告日志;

访问master:http://ip:8080

5.提交python作业:

bin/spark-submit wordcount.py

6.提交jar作业:

bin/spark-submit --class com.test.spark.SparkTest --num-executors 3 --driver-memory 512M --executor-memory 512M --executor-cores 1 --master spark://hostname:7077 /home/guest/bigdata/spark-2.3.0-bin-hadoop2.7/test.jar

猜你喜欢

转载自blog.csdn.net/huashetianzu/article/details/83243681