Host: 192.168.0.135 192.168.0.136 192.168.0.137
master: 137 workers:135 136
1.Install spark on all hosts in /opt dir
2.Install SSH Remote Access
137#ssh-keygen
137#ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
137#ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
3. Configure
a. master 137
slaves
192.168.0.135 192.168.0.136
spark-env.sh
export SPARK_MASTER_IP=192.168.0.137 export SPARK_MASTER_PORT=7077
b. cp spark-env.sh to all workers nodes
4. start clusters
137#./sbin/start-all.sh
access: http://192.168.0.137:8080/ http://192.168.0.136:8081/
References
https://spark.apache.org/docs/latest/spark-standalone.html
https://trongkhoanguyenblog.wordpress.com/2014/11/15/how-to-install-spark-1-1-on-ubuntu/