Hadoop的Job执行MR的时候抛AccessControlException异常,详情如下:
org.apache.hadoop.hdfs.DFSClient: Short circuit access failed org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Can't continue with getBlockLocalPathInfo() authorization. The user XXX is not allowed to call getBlockLocalPathInfo at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
产生问题的原因定位:
org.apache.hadoop.hdfs.server.datanode.DataNode; public BlockLocalPathInfo getBlockLocalPathInfo(Block block, Token<BlockTokenIdentifier> token) throws IOException ;
具体的执行如下:
private boolean shouldTryShortCircuitRead(InetSocketAddress targetAddr) throws IOException { if (shortCircuitLocalReads && isLocalAddress(targetAddr)) { return true; } return false; }
1.读取DFSConfigKeys.DFS_CLIENT_READ_SHORTCIRCUIT_KEY对应的值如果是true那么走getLocalBlockReader(...)如果是false那么走getBlockReader(...)
2.如果走getLocalBlockReader(...)那么需要通过BlockReaderLocal.newBlockReader(...)创建BlockReader
3.执行getBlockPathInfo(...)的时候会执行DataNode的getBlockLocalPathInfo方法从而在checkBlockLocalPathAccess的时候告知权限不足.
解决方式:
1.直接禁用
conf.setBoolean(DFSConfigKeys.DFS_CLIENT_READ_SHORTCIRCUIT_KEY, false);