参考官方文档逐步安装部暑,ambari 2.7 上 部署hadoop 3.0,一些错误解决笑记
Ambari 2.7 installation guide
网络相关配置
互信
ssh-keygen // 一路回车生成公钥
ssh-copy-id <target host> //复制到目标主机
systemctl enable ntpd //启动ntp
systemctl disable firewalld //关闭防火墙
service firewalld stop
setenforce 0
echo umask 0022 >> /etc/profile
org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host
YARN Registry DNS Start failed
ERROR dns.PrivilegedRegistryDNSStarter (PrivilegedRegistryDNSStarter.java:init(61)) - Error initializing Registry DNS
java.net.BindException: Problem binding to [hadoop-app1.novalocal:53] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:736)
at org.apache.hadoop.registry.server.dns.RegistryDNS.openUDPChannel(RegistryDNS.java:1016)
at org.apache.hadoop.registry.server.dns.RegistryDNS.addNIOUDP(RegistryDNS.java:925)
at org.apache.hadoop.registry.server.dns.RegistryDNS.initializeChannels(RegistryDNS.java:196)
at
解决方法——————
参考:https://community.hortonworks.com/questions/224670/yarn-registry-dns-start-failed-hortonworks-3.html
YARN 设置命令 https://yarnpkg.com/lang/en/docs/cli/config/
错误原因:
安装YARN的HOST上,PORT 53 已被占用
可以使用 netstat -tnlpa | grep 53 命令查看
[root@hadoop-app1 ~]# netstat -tnlpa | grep 53
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 1307/dnsmasq
tcp 0 0 10.0.14.134:35534 10.0.14.134:53 TIME_WAIT -
tcp 0 0 10.0.14.134:52538 10.0.10.94:22 TIME_WAIT -
进入YARN 设置文件重新设置端口 vi /usr/hdp/3.0.1.0-187/hadoop-yarn/conf/yarn-site.xml
修改保存后用命令启动 registrydns yarn --daemon start registrydns
<property>
<name>hadoop.registry.dns.bind-port</name>
<value>53</value> //改成其他端口如 54
<hidden>true</hidden>
</property>
然后在ambari UI 管理 yarn - start service
这个操作有个头疼的问题,如果在ambari web选择 restart service似乎端口又会改成53,然后。。。又failed
所以把53端口的进程KILL了,再启动,似乎更实用点。。
[root@hadoop-app1 bin]# sudo lsof -i -P -n |grep 53
java 680 atlas 305u IPv6 765376 0t0 TCP 10.0.14.134:48446->10.0.11.117:2181 (ESTABLISHED)
avahi-dae 2123 avahi 12u IPv4 18340 0t0 UDP *:5353
dnsmasq 2950 nobody 5u IPv4 29192 0t0 UDP 192.168.122.1:53
dnsmasq 2950 nobody 6u IPv4 29193 0t0 TCP 192.168.122.1:53 (LISTEN)
java 8454 druid 535u IPv6 799296 0t0 TCP 10.0.14.134:49518->10.0.11.117:2181 (ESTABLISHED)
java 8939 druid 589u IPv6 904539 0t0 TCP 10.0.14.134:8081->10.0.14.134:45014 (ESTABLISHED)
java 8939 druid 593u IPv6 905141 0t0 TCP 10.0.14.134:8081->10.0.14.134:45364 (ESTABLISHED)
java 9792 druid 427u IPv6 917355 0t0 TCP 10.0.14.134:45364->10.0.14.134:8081 (ESTABLISHED)
java 12382 hdfs 351u IPv4 918492 0t0 TCP 10.0.14.134:8020->10.0.14.115:53366 (ESTABLISHED)
java 12382 hdfs 396u IPv4 918489 0t0 TCP 10.0.14.134:8020-
....
[root@hadoop-app1 bin]# kill -9 2950 // 把那个nobody 又占用端口53的PID2958 kill 了
启动后YARN不再报错
==============================================
accumulo 启动失败 var/log/accumulo/accumulo-init.err' returned 255
Traceback (most recent call last):
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart
self.status(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 75, in status
check_process_status(pid_file)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
raise ComponentIsNotRunning()
ComponentIsNotRunning
The above exception was the cause of the following exception:
解决参考:https://community.hortonworks.com/articles/79327/re-initializing-apache-accumulo-under-hdp.html
确保 ZOOKEEPER 已启动
在accumulo 安装HOST上运行
sudo -u hdfs hdfs dfs -rm -R /apps/accumulo/data
sudo -u accumulo ACCUMULO_CONF_DIR=/etc/accumulo/conf/server accumulo init --instance-name hdp-accumulo-instance --clear-instance-name
==============================================
mysql-jdbc 安装
- sudo yum install mysql-connector-java*
- ls -al /usr/share/java/mysql-connector-java.jar
- cd /var/lib/ambari-server/resources/
ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar