1.startup command
./bin/spark-submit --class org.apache.spark.examples.JavaWordCount --deploy-mode client --master spark://gzsw-02:7077 lib/spark-examples-1.4.1-hadoop2.4.0.jar hdfs://host02:/user/hadoop/input.txt
note:1) the master is the cluster manager which stated in spark master ui page,ie,
URL: spark://gzsw-02:7077
other than the 'REST URL xxxx' .
2) the --deploy-mode is optional,ie it's the same effect as specified as above 'client'.
2.run logs
Spark Command: /usr/local/jdk/jdk1.6.0_31/bin/java -cp /home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/conf/:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/spark-assembly-1.4.1-hadoop2.4.0.jar:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/usr/local/hadoop/hadoop-2.5.2/etc/hadoop/ -Xms6g -Xmx6g -XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --master spark://gzsw-02:7077 --deploy-mode client --class org.apache.spark.examples.JavaWordCount lib/spark-examples-1.4.1-hadoop2.4.0.jar hdfs://hd02:/user/hadoop/input.txt ======================================== -executed cmd retruned by Main.java:/usr/local/jdk/jdk1.6.0_31/bin/java -cp /home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/conf/:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/spark-assembly-1.4.1-hadoop2.4.0.jar:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/usr/local/hadoop/hadoop-2.5.2/etc/hadoop/ -Xms6g -Xmx6g -XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --master spark://gzsw-02:7077 --deploy-mode client --class org.apache.spark.examples.JavaWordCount lib/spark-examples-1.4.1-hadoop2.4.0.jar hdfs://hd02:/user/hadoop/input.txt 16/09/19 11:28:17 INFO spark.SparkContext: Running Spark version 1.4.1 16/09/19 11:28:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/19 11:28:18 INFO spark.SecurityManager: Changing view acls to: hadoop 16/09/19 11:28:18 INFO spark.SecurityManager: Changing modify acls to: hadoop 16/09/19 11:28:18 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop) 16/09/19 11:28:19 INFO slf4j.Slf4jLogger: Slf4jLogger started 16/09/19 11:28:19 INFO Remoting: Starting remoting 16/09/19 11:28:19 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:55817] 16/09/19 11:28:19 INFO util.Utils: Successfully started service 'sparkDriver' on port 55817. 16/09/19 11:28:19 INFO spark.SparkEnv: Registering MapOutputTracker 16/09/19 11:28:19 INFO spark.SparkEnv: Registering BlockManagerMaster 16/09/19 11:28:19 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-e19f684c-686b-4841-a863-2e143face3c3/blockmgr-b6ea03ec-ee30-4133-992f-ccf27ef35c93 16/09/19 11:28:19 INFO storage.MemoryStore: MemoryStore started with capacity 2.6 GB 16/09/19 11:28:19 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-e19f684c-686b-4841-a863-2e143face3c3/httpd-1c0a5a85-eef0-4e19-a244-ca7a9c852204 16/09/19 11:28:19 INFO spark.HttpServer: Starting HTTP Server 16/09/19 11:28:19 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/09/19 11:28:19 INFO server.AbstractConnector: Started [email protected]:48267 16/09/19 11:28:19 INFO util.Utils: Successfully started service 'HTTP file server' on port 48267. 16/09/19 11:28:19 INFO spark.SparkEnv: Registering OutputCommitCoordinator 16/09/19 11:28:19 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/09/19 11:28:19 INFO server.AbstractConnector: Started [email protected]:7106 16/09/19 11:28:19 INFO util.Utils: Successfully started service 'SparkUI' on port 7106. 16/09/19 11:28:19 INFO ui.SparkUI: Started SparkUI at http://192.168.100.4:7106 16/09/19 11:28:19 INFO spark.SparkContext: Added JAR file:/home/hadoop/spark/spark-1.4.1-bin-hadoop2.4/lib/spark-examples-1.4.1-hadoop2.4.0.jar at http://192.168.100.4:48267/jars/spark-examples-1.4.1-hadoop2.4.0.jar with timestamp 1474255699921 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@gzsw-02:7077/user/Master... 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160919112820-0002 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/0 on worker-20160914175458-192.168.100.15-36198 (192.168.100.15:36198) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/0 on hostPort 192.168.100.15:36198 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/1 on worker-20160914175458-192.168.100.15-36198 (192.168.100.15:36198) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/1 on hostPort 192.168.100.15:36198 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/2 on worker-20160914175457-192.168.100.11-41800 (192.168.100.11:41800) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/2 on hostPort 192.168.100.11:41800 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/3 on worker-20160914175457-192.168.100.11-41800 (192.168.100.11:41800) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/3 on hostPort 192.168.100.11:41800 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/4 on worker-20160914175457-192.168.100.10-46154 (192.168.100.10:46154) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/4 on hostPort 192.168.100.10:46154 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/5 on worker-20160914175457-192.168.100.10-46154 (192.168.100.10:46154) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/5 on hostPort 192.168.100.10:46154 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/6 on worker-20160914175457-192.168.100.6-51383 (192.168.100.6:51383) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/6 on hostPort 192.168.100.6:51383 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/7 on worker-20160914175457-192.168.100.6-51383 (192.168.100.6:51383) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/7 on hostPort 192.168.100.6:51383 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/8 on worker-20160914175457-192.168.100.9-50567 (192.168.100.9:50567) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/8 on hostPort 192.168.100.9:50567 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/9 on worker-20160914175457-192.168.100.9-50567 (192.168.100.9:50567) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/9 on hostPort 192.168.100.9:50567 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/10 on worker-20160914175456-192.168.100.7-36541 (192.168.100.7:36541) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/10 on hostPort 192.168.100.7:36541 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/11 on worker-20160914175456-192.168.100.7-36541 (192.168.100.7:36541) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/11 on hostPort 192.168.100.7:36541 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/12 on worker-20160914175457-192.168.100.8-38650 (192.168.100.8:38650) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/12 on hostPort 192.168.100.8:38650 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/13 on worker-20160914175457-192.168.100.8-38650 (192.168.100.8:38650) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/13 on hostPort 192.168.100.8:38650 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/14 on worker-20160914175457-192.168.100.13-43911 (192.168.100.13:43911) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/14 on hostPort 192.168.100.13:43911 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/15 on worker-20160914175457-192.168.100.13-43911 (192.168.100.13:43911) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/15 on hostPort 192.168.100.13:43911 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/16 on worker-20160914175457-192.168.100.12-44199 (192.168.100.12:44199) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/16 on hostPort 192.168.100.12:44199 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/17 on worker-20160914175457-192.168.100.12-44199 (192.168.100.12:44199) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/17 on hostPort 192.168.100.12:44199 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/18 on worker-20160914175456-192.168.100.14-36693 (192.168.100.14:36693) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/18 on hostPort 192.168.100.14:36693 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor added: app-20160919112820-0002/19 on worker-20160914175456-192.168.100.14-36693 (192.168.100.14:36693) with 2 cores 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160919112820-0002/19 on hostPort 192.168.100.14:36693 with 2 cores, 2.0 GB RAM 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/0 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/2 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/4 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/1 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/3 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/6 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/5 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/7 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/10 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/8 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/9 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/12 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/11 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/13 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/14 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/16 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/15 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/17 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/18 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/0 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/19 is now LOADING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/1 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/2 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/3 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/4 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/5 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/6 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/7 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/8 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/9 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/10 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/11 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/12 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/13 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/14 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/15 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/16 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/17 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/18 is now RUNNING 16/09/19 11:28:20 INFO client.AppClient$ClientActor: Executor updated: app-20160919112820-0002/19 is now RUNNING 16/09/19 11:28:20 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38299. 16/09/19 11:28:20 INFO netty.NettyBlockTransferService: Server created on 38299 16/09/19 11:28:20 INFO storage.BlockManagerMaster: Trying to register BlockManager 16/09/19 11:28:20 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.4:38299 with 2.6 GB RAM, BlockManagerId(driver, 192.168.100.4, 38299) 16/09/19 11:28:20 INFO storage.BlockManagerMaster: Registered BlockManager 16/09/19 11:28:20 INFO scheduler.EventLoggingListener: Logging events to file:/home/hadoop/spark/spark-eventlog/app-20160919112820-0002 16/09/19 11:28:20 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 16/09/19 11:28:21 INFO storage.MemoryStore: ensureFreeSpace(228680) called with curMem=0, maxMem=2778306969 16/09/19 11:28:21 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 223.3 KB, free 2.6 GB) 16/09/19 11:28:21 INFO storage.MemoryStore: ensureFreeSpace(18130) called with curMem=228680, maxMem=2778306969 16/09/19 11:28:21 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 17.7 KB, free 2.6 GB) 16/09/19 11:28:21 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.100.4:38299 (size: 17.7 KB, free: 2.6 GB) 16/09/19 11:28:21 INFO spark.SparkContext: Created broadcast 0 from textFile at JavaWordCount.java:45 16/09/19 11:28:21 INFO mapred.FileInputFormat: Total input paths to process : 1 16/09/19 11:28:21 INFO spark.SparkContext: Starting job: collect at JavaWordCount.java:68 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Registering RDD 3 (mapToPair at JavaWordCount.java:54) 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Got job 0 (collect at JavaWordCount.java:68) with 1 output partitions (allowLocal=false) 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Final stage: ResultStage 1(collect at JavaWordCount.java:68) 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 0) 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at JavaWordCount.java:54), which has no missing parents 16/09/19 11:28:21 INFO storage.MemoryStore: ensureFreeSpace(4760) called with curMem=246810, maxMem=2778306969 16/09/19 11:28:21 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.6 KB, free 2.6 GB) 16/09/19 11:28:21 INFO storage.MemoryStore: ensureFreeSpace(2666) called with curMem=251570, maxMem=2778306969 16/09/19 11:28:21 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.6 KB, free 2.6 GB) 16/09/19 11:28:21 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.100.4:38299 (size: 2.6 KB, free: 2.6 GB) 16/09/19 11:28:21 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874 16/09/19 11:28:21 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at JavaWordCount.java:54) 16/09/19 11:28:21 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:32872/user/Executor#-695323121]) with ID 19 16/09/19 11:28:23 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.100.14, ANY, 1474 bytes) 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:34254/user/Executor#680848952]) with ID 4 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:45597/user/Executor#61882158]) with ID 17 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:55846/user/Executor#-109266911]) with ID 2 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:39056/user/Executor#-730178427]) with ID 3 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:41904/user/Executor#172197607]) with ID 7 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:48188/user/Executor#1126474595]) with ID 1 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:53231/user/Executor#643650421]) with ID 0 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:44498/user/Executor#1819346495]) with ID 10 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:33380/user/Executor#1519517929]) with ID 5 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:51226/user/Executor#107130314]) with ID 14 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:51287/user/Executor#-786570003]) with ID 18 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:42736/user/Executor#-1654106840]) with ID 13 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:46813/user/Executor#-1544202525]) with ID 12 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:59695/user/Executor#758748544]) with ID 15 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:32978/user/Executor#1666271797]) with ID 9 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:43395/user/Executor#1432829817]) with ID 8 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:55244/user/Executor#-583314465]) with ID 6 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:36554/user/Executor#464137168]) with ID 11 16/09/19 11:28:23 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:51715/user/Executor#1409392060]) with ID 16 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.14:41531 with 906.2 MB RAM, BlockManagerId(19, 192.168.100.14, 41531) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.10:55997 with 906.2 MB RAM, BlockManagerId(4, 192.168.100.10, 55997) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.12:39237 with 906.2 MB RAM, BlockManagerId(17, 192.168.100.12, 39237) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.11:55993 with 906.2 MB RAM, BlockManagerId(2, 192.168.100.11, 55993) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.10:50592 with 906.2 MB RAM, BlockManagerId(5, 192.168.100.10, 50592) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.6:53362 with 906.2 MB RAM, BlockManagerId(7, 192.168.100.6, 53362) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.11:38102 with 906.2 MB RAM, BlockManagerId(3, 192.168.100.11, 38102) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.7:36601 with 906.2 MB RAM, BlockManagerId(10, 192.168.100.7, 36601) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.14:40729 with 906.2 MB RAM, BlockManagerId(18, 192.168.100.14, 40729) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.8:53863 with 906.2 MB RAM, BlockManagerId(13, 192.168.100.8, 53863) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.13:43366 with 906.2 MB RAM, BlockManagerId(14, 192.168.100.13, 43366) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.9:54016 with 906.2 MB RAM, BlockManagerId(9, 192.168.100.9, 54016) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.6:60357 with 906.2 MB RAM, BlockManagerId(6, 192.168.100.6, 60357) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.9:47909 with 906.2 MB RAM, BlockManagerId(8, 192.168.100.9, 47909) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.15:46824 with 906.2 MB RAM, BlockManagerId(1, 192.168.100.15, 46824) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.15:48343 with 906.2 MB RAM, BlockManagerId(0, 192.168.100.15, 48343) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.8:32827 with 906.2 MB RAM, BlockManagerId(12, 192.168.100.8, 32827) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.13:52959 with 906.2 MB RAM, BlockManagerId(15, 192.168.100.13, 52959) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.7:33396 with 906.2 MB RAM, BlockManagerId(11, 192.168.100.7, 33396) 16/09/19 11:28:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.100.12:59404 with 906.2 MB RAM, BlockManagerId(16, 192.168.100.12, 59404) 16/09/19 11:28:25 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.100.14:41531 (size: 2.6 KB, free: 906.2 MB) 16/09/19 11:28:25 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.100.14:41531 (size: 17.7 KB, free: 906.2 MB) 16/09/19 11:28:26 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3398 ms on 192.168.100.14 (1/1) 16/09/19 11:28:26 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 16/09/19 11:28:26 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (mapToPair at JavaWordCount.java:54) finished in 4.501 s 16/09/19 11:28:26 INFO scheduler.DAGScheduler: looking for newly runnable stages 16/09/19 11:28:26 INFO scheduler.DAGScheduler: running: Set() 16/09/19 11:28:26 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1) 16/09/19 11:28:26 INFO scheduler.DAGScheduler: failed: Set() 16/09/19 11:28:26 INFO scheduler.DAGScheduler: Missing parents for ResultStage 1: List() 16/09/19 11:28:26 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at JavaWordCount.java:61), which is now runnable 16/09/19 11:28:26 INFO storage.MemoryStore: ensureFreeSpace(2408) called with curMem=254236, maxMem=2778306969 16/09/19 11:28:26 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.4 KB, free 2.6 GB) 16/09/19 11:28:26 INFO storage.MemoryStore: ensureFreeSpace(1458) called with curMem=256644, maxMem=2778306969 16/09/19 11:28:26 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1458.0 B, free 2.6 GB) 16/09/19 11:28:26 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.100.4:38299 (size: 1458.0 B, free: 2.6 GB) 16/09/19 11:28:26 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874 16/09/19 11:28:26 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at JavaWordCount.java:61) 16/09/19 11:28:26 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks 16/09/19 11:28:26 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, 192.168.100.15, PROCESS_LOCAL, 1243 bytes) 16/09/19 11:28:28 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.100.4:38299 in memory (size: 2.6 KB, free: 2.6 GB) 16/09/19 11:28:28 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.100.14:41531 in memory (size: 2.6 KB, free: 906.2 MB) 16/09/19 11:28:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.100.15:46824 (size: 1458.0 B, free: 906.2 MB) 16/09/19 11:28:28 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.100.15:48188 16/09/19 11:28:28 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 144 bytes 16/09/19 11:28:28 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 2322 ms on 192.168.100.15 (1/1) 16/09/19 11:28:28 INFO scheduler.DAGScheduler: ResultStage 1 (collect at JavaWordCount.java:68) finished in 2.322 s 16/09/19 11:28:28 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 16/09/19 11:28:28 INFO scheduler.DAGScheduler: Job 0 finished: collect at JavaWordCount.java:68, took 6.953234 s are: 1 back: 1 is: 3 ERROR: 1 a: 2 on: 1 content: 2 bad: 2 with: 1 some: 1 INFO: 4 to: 1 : 2 This: 3 more: 1 message: 1 More: 1 thing: 1 warning: 1 WARN: 2 normal: 1 Something: 1 happened: 1 other: 1 messages: 2 details: 1 the: 1 Here: 1 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 16/09/19 11:28:28 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 16/09/19 11:28:28 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.100.4:7106 16/09/19 11:28:28 INFO scheduler.DAGScheduler: Stopping DAGScheduler 16/09/19 11:28:28 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors 16/09/19 11:28:28 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down 16/09/19 11:28:29 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/09/19 11:28:29 INFO util.Utils: path = /tmp/spark-e19f684c-686b-4841-a863-2e143face3c3/blockmgr-b6ea03ec-ee30-4133-992f-ccf27ef35c93, already present as root for deletion. 16/09/19 11:28:29 INFO storage.MemoryStore: MemoryStore cleared 16/09/19 11:28:29 INFO storage.BlockManager: BlockManager stopped 16/09/19 11:28:29 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 16/09/19 11:28:29 INFO spark.SparkContext: Successfully stopped SparkContext 16/09/19 11:28:29 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/09/19 11:28:29 INFO util.Utils: Shutdown hook called 16/09/19 11:28:29 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/09/19 11:28:29 INFO util.Utils: Deleting directory /tmp/spark-e19f684c-686b-4841-a863-2e143face3c3 16/09/19 11:28:29 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/09/19 11:28:29 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
so we know the driver is run on local host(192.168.100.4:55817).