org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.

问题背景

自己机器上搭建的集群(apache体系),使用hive(采用Tez作为执行引擎)执行hql时出现的错误。
详细如下:

org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. 
This token is expired. current time is 1574768024902 found 1574765572175
Note: System times on machines may be out of sync. Check system time and time zones.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.tez.dag.app.launcher.TezContainerLauncherImpl$Container.launch(TezContainerLauncherImpl.java:171)
        at org.apache.tez.dag.app.launcher.TezContainerLauncherImpl$EventProcessor.run(TezContainerLauncherImpl.java:396)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

关键信息:

System times on machines may be out of sync. Check system time and time zones.

解决

看样子是集群时钟同步问题,date命令看一下果然如此;之前关闭了ntpd服务。配置ntpd时钟同步服务或手动设置时间都可。

这里自己写了个脚本以方便配置(dateSets.sh)

#!/bin/bash
#dateSets.sh
for host in chdp11 chdp12 chdp13
do
        echo "========== $host =========="
        ssh -t $host "date -s $1"
done

在这里插入图片描述
最后很奇怪是,在本次开启的hive执行窗口下,前面几次执行hql都是正常的,啥都没改就出现这个问题了。如果集群时钟一开始就没有同步,不应该是一开始就无法执行吗?或许是和Tez session有关系?

发布了130 篇原创文章 · 获赞 39 · 访问量 1万+

猜你喜欢

转载自blog.csdn.net/qq_34901049/article/details/103997020