Spark-shell

问题：Spark-shell启动时报错WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

解决方案：
在/etc/profile设置一下：export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
在spark的conf/spark-env.sh文件加入：export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native

Spark-submit

问题：提交任务到yarn时报WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

解决方案：
在hdfs上创建目录

hdfs dfs -mkdir /spark_jars

上传spark的所有jar包到hdfs的/spark_jars目录

hdfs dfs -put /usr/local/spark/jars/* /spark_jars/ 

在${SPARK_HOME}/conf/spark-defaults.conf文件中新增如下配置：

spark.yarn.jars			hdfs://hadoopSvr1:8020/spark-jars/*

问题：spark 与 Hadoop 融合后提交作业 slf4j提示Class path contains multiple SLF4J bindings

具体WARN信息如下：

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

解决方案：
主要原因是spark与hadoop融合后，依赖冲突了，删掉其中一个jar包(或改名)即可（下面以改名spark下的jar包为例）

cd /usr/local/spark/jars/
mv slf4j-log4j12-1.7.16.jar slf4j-log4j12-1.7.16.jar.backup

具体可参考：https://www.slf4j.org/codes.html#multiple_bindings