- 准备环境
- 安装JDK, 不需要安装Scala, Spark内置scala, 最好搭配JDK8, 搭配8以上版本, 依据个人经验可能出问题。
- 上传spark安装包
- 解压spark并修改配置文件(两个配置文件,第一个配置文件添加了3个配置文件)
Spark.env.sh
文件,增加如下3个属性, 指定JDK, 指定Spark master, 指定Spark集群rpc通讯端口
export JAVA_HOME=/root/apps/jdk1.8.0_65
export SPARK_MASTER_HOST=hadoop-01
export SPARK_MASTER_PORT=7077
slaves
文件
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# A Spark Worker will be started on each of the machines listed below.
hadoop-03
hadoop-04
hadoop-05
- 将配置好的spark安装程序拷贝给其他机器, 可以写一个循环的shell
for i in {3..5};
do
scp -r /root/apps/spark-2.1.1-bin-hadoop2.7/ root@hadoop-0$i:/root/apps/;
done
- 启动spark (sbin/start-all.sh)
Worker通过读取spark-env.sh
文件得知Master在哪里的 - 通过web页面访问spark管理页面(master所在机器的地址+8080端口), 注意: 此8080并非tomcat或者啥乱七八糟的, 而是netty通讯端口。