测试要求:
三台物理机
内存:64 G
查看物理CPU个数:2
查看每个物理CPU中core的个数(即核数) : 12
一、准备测试环境
1.下载软件并安装
-
BigDataBench_V4.0_Spark.tar.gz http://prof.ict.ac.cn/download.html#Full_download
-
Apache Spark 集群
-
Hadoop 集群
-
Scala
二、环境部署及测试常规任务
此处只介绍BigDataBench_V4.0_Spark安装部署,spark、hadoop、scala略
1. 解压压缩包
tar -zxvf BigDataBench_V4.0_Spark.tar.gz
2.切换目录
cd BigDataBench_V4.0_Spark
3.配置资源文件
vi conf.properties
配置相关资源
##
export BigdataBench_HOMESP=/opt/rdma/BigDataBench_V4.0_Spark
##JAVA
export JAVA_HOME=/opt/sensing/datafly/deps/jdk/jdk1.8.0_60
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar
export JRE_HOME=$JAVA_HOME/jre
##HADOOP
export HADOOP_HOME=/opt/hadoop-2.7.3
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LIB=${HADOOP_HOME}/lib
export CLASSPATH=$HADOOP_HOME/hadoop-core-*.jar:$CLASSPATH
##SPARK
export SCALA_HOME=/opt/scala-2.11.8
export PATH=$PATH:$SCALA_HOME/bin
export SPARK_HOME=/opt/spark-2.2.0
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
export JAR_FILE=$BigdataBench_HOMESP/JAR_FILE
下载 http://mirrors.ustc.edu.cn/gnu/gsl/gsl-2.4.tar.gz备注:中间可能需要安装gsl,安装gsl-2.4 即可
安装步骤:
-
解压缩 :tar -zxvf gsl-2.4.tar.gz
-
./configure --prefix=/opt/gsl-2.4
-
make
-
make install
4.生成测试数据
cd MicroBenchmark/OfflineAnalytics
[root@RDP1 OfflineAnalytics]# ls
FFT Grep Matrix MD5 RandSample Sort WordCount
生成WordCount测试数据
cd WordCount
sh genData_WordCount.sh
输入要生成数据的数据大小(单位GB)
数据会提交到HDFS 上/opt/hadoop-2.7.3/bin/hadoop dfs -ls /spark/wordcount/data 目录
查看数据:
/opt/hadoop-2.7.3/bin/hadoop dfs -ls /spark/wordcount/data
执行
sh run_WordCount.sh
三、配置RDMA
1.安装RDMA Packages 集群机器全部一样执行
#查看所需包
yum groupinfo "Infiniband Support"
#安装强制需要的包
yum -y groupinstall "Infiniband Support"
2.启用RDMA
systemctl start rdma.service
3.查看RDMA状态
systemctl status rdma.service
四、更改Spark 配置更改为RDMA
1.下载SparkRDMA相关依赖包及so文件
https://github.com/Mellanox/SparkRDMA/releases/download/3.0/spark-rdma-3.0.tgz
下载完成上传到/opt/spark-rdma/spark-rdma-3.0.tgz 目录,集群环境节点均上传
此处依赖jar包可以自定义根据spark版本下载Maven项目自己打包,也可以直接下载,目前支持 2.1.0, 2.2.0 or 2.3.0 三个版本
2.配置.so文件
我们需要将 libdisni.so 文件分发到集群的所有节点的同一目录下,然后配置下面的环境:
#export lib
export C_INCLUDE_PATH=$C_INCLUDE_PATH:/opt/gsl-2.4/include
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/opt/gsl-2.4/include
export LD_LIBRARY_PATH=/opt/gsl-2.4/lib
export LIBRARY_PATH=/opt/gsl-2.4/lib
3.安装其他依赖
其中 /home/iteblog/spark-2.1.0-bin/rdma/ 存放了libdisni.so 文件。运行的过程中可能还需要 libibverbs.so.1 和 librdmacm.so.1 文件,可以通过下面命令解决
yum -y install libibverbs librdmacm
4. 将 SparkRDMA 模块加入到 Spark 的运行环境中去
spark.driver.extraClassPath /path/to/SparkRDMA/spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
spark.executor.extraClassPath /path/to/SparkRDMA/spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
5.更改Shuffle Manager
$SPARK_HOME/conf/spark-defaults.conf下更改shuffle Manager
spark.shuffle.manager org.apache.spark.shuffle.rdma.RdmaShuffleManager
6.以RDMA模式启动spark
bin/spark-shell --master yarn-client --driver-memory 5g --executor-memory 10g \
--queue rdma --executor-cores 1 --num-executors 3 \
--conf "spark.yarn.dist.archives=/opt/spark-rdma/spark-rdma-3.0.tgz" \
--conf "spark.executor.extraLibraryPath=/opt/gsl-2.4/lib/libdisni.so" \
--conf "spark.driver.extraLibraryPath=/opt/gsl-2.4/lib/libdisni.so" \
--conf "spark.executor.extraClassPath=rdma.tgz/rdma/*" \
--conf "spark.driver.extraClassPath=/opt/gsl-2.4/lib/*" \
--conf "spark.shuffle.manager=org.apache.spark.shuffle.rdma.RdmaShuffleManager"
报此异常显示不支持RDMA,需要单独网卡和配置
五、参考资料
RDMA源码:
https://github.com/Mellanox/SparkRDMA
配置RDMA:
https://www.rdmamojo.com/2014/10/11/working-rdma-redhatcentos-7/
性能监控使用 Nmon:
https://blog.csdn.net/u012587561/article/details/52166851