hadoop3完全分布式搭建可参照:https://blog.csdn.net/qq_33931272/article/details/79884656
一些问题:
1、ssh条件:
1)注意关闭防火墙;
2)注意.ssh文件的权限;
3)注意.ssh文件是在hadoop下生成的,而不是root下
4)静态的IP地址
2、关于hadoop3的配置
1、如果jdk用的是9以上需要在配置的配置 yarn-env.sh中添加以下内容,因为jdk9中多了一些限制
export YARN_RESOURCEMANAGER_OPTS="--add-modules=ALL-SYSTEM" export YARN_NODEMANAGER_OPTS="--add-modules=ALL-SYSTEM错误如下:
1) Error injecting constructor, java.lang.NoClassDefFoundError: javax/activation/DataSource at org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:52) at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer$NMWebApp.setup(WebServer.java:153) while locating org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver
2、有时候datanote会运行不起来这就需要去将hdfs中的data文件删掉
错误如下:
错误datanode: 2018-04-11 05:09:53,773 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid 5bdd3f47-9d7d-489c-8702-b0ede70da713) service to slave1/192.168.8.131:9000. Exiting. java.io.IOException: All specified directories have failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:552) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1705) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1665) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:390) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816) at java.base/java.lang.Thread.run(Thread.java:844)则需要:
rm -r /usr/hadoop/hdfs/data
3、hadoop3中端口有改变
4、配置文件:workers(不再是slaves)
将内容改为:
- slave1
- slave2