supplement to last blog :https://blog.csdn.net/yao09605/article/details/103971966
solving the WARN message:
Neither spark.yarn.jars nor spark.yarn.archive is set
$ jar -cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
$ dfs -mkdir /system
$ dfs -mkdir /system/SparkJars
$ dfs -mkdir /system/SparkJars/jar
$ hdfs dfs -put spark-libs.jar /system/SparkJars/jar
$ cd $SPARK_HOME/conf
$ cp spark-defaults.conf.template spark-defauls.conf
add
spark.yarn.archive=hdfs:///system/SparkJars/jar/spark-libs.jar
run
$ spark-submit --master yarn app.py
should be ok by now