$ startspark
org.apache.spark.deploy.master.Master running as process 1067300. Stop it first.
Desktop: org.apache.spark.deploy.worker.Worker running as process 1067817. Stop it first.
Laptop: starting org.apache.spark.deploy.worker.Worker, logging to /home/appleyuchi/bigdata/spark-3.0.0-bin-hadoop3.2/logs/spark-appleyuchi-org.apache.spark.deploy.worker.Worker-1-Laptop.out
Laptop: failed to launch: nice -n 0 /home/appleyuchi/bigdata/spark-3.0.0-bin-hadoop3.2/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://Desktop:7077
Laptop: at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
Laptop: at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
Laptop: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
Laptop: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
Laptop: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
Laptop: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
Laptop: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Laptop: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Laptop: at java.lang.Thread.run(Thread.java:748)
Laptop: 20/07/30 15:35:03 INFO ShutdownHookManager: Shutdown hook called
Laptop: full log in /home/appleyuchi/bigdata/spark-3.0.0-bin-hadoop3.2/logs/spark-appleyuchi-org.apache.spark.deploy.worker.Worker-1-Laptop.out
启动spark集群失败,想起来自己更新过spark集群
cat spark-appleyuchi-org.apache.spark.deploy.worker.Worker-1-Laptop.out
20/07/30 15:35:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(appleyuchi); groups with view permissions: Set(); users with modify permissions: Set(appleyuchi); groups with modify permissions: Set()
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 WARN Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
20/07/30 15:35:02 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
解决方案:
Laptop节点的$SPARK_HOME/conf/spark-env.sh
export SPARK_LOCAL_IP=Laptop
每个节点的这个SPARK_LOCAL_IP变量都必须是节点所在的局域网的域名或者IP
然后重启spark集群报错消失。