DevOps(6)Spark Deployment on VM 2
1. Workers Sync from Master
under SPARK_HOME/conf/spark-env.sh
It works in cluster mode.
#SCALA_HOME=/opt/scala/scala-2.10.3
export SPARK_WORKER_MEMORY=768m
#SPARK_CLASSPATH='/opt/profiles-spark/lib/*.jar'
export SPARK_JAVA_OPTS="-Dbuild.env=sparkvm"
#export SPARK_DAEMON_JAVA_OPTS="-Dbuild.env=sparkvm"
#jobJar='/opt/profiles-spark/lib/*.jar'
#export JAVA_OPTS="-Dbuild.env=sparkvm"
export USER=carl
It requires we extend the serialization interface.
2. Pre-place the binary files
Did not try that. I do not like this way.
References: