版权声明:本文为博主原创文章,如需转载,请注明出处。 https://blog.csdn.net/Post_Yuan/article/details/81286792
现象
在做大表Load数据的时候遇到8448错误,具有错误信息如下,
ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::doBulkLoad returned error HBASE_DOBULK_LOAD_ERROR(-715). Cause: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=TRAF_RSRVD_3_TRAFODION.SEABASE.CQGPRS20180730_SNAPSHOT table=TRAF_RSRVD_3:TRAFODION.SEABASE.CQGPRS20180730 type=FLUSH } had an error. Procedure TRAF_RSRVD_3_TRAFODION.SEABASE.CQGPRS20180730_SNAPSHOT { waiting=[] done=[swrzdn162.cqyd.com,60020,1532924595692, swrzdn123.cqyd.com,60020,1532927440461, swrzdn02.cqyd.com,60020,1532921839386, swrzdn163.cqyd.com,60020,1532929048293, swrzdn124.cqyd.com,60020,1530530102339, swrzdn122.cqyd.com,60020,1532923397158, swrzdn161.cqyd.com,60020,1530530101720, swrzdn121.cqyd.com,60020,1530530102247, swrzdn125.cqyd.com,60020,1530530101579, swrzdn159.cqyd.com,60020,1530530102111, swrzdn03.cqyd.com,60020,1532925830807, swrzdn164.cqyd.com,60020,1530530101723, swrzdn128.cqyd.com,60020,1530530101803, swrzdn110.cqyd.com,60020,1530530102367] }
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:368)
at org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1022)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55714)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via swrzdn124.cqyd.com,60020,1530530102339:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.NotServingRegionException
分析
通过以上错误信息,发现报错与Snapshot有关,由于目标表较大,在对表做Snapshot的时间超出了配置的timeout设置,导致失败。
解决
调整HBase配置中与Snapshot相关的两个timeout配置,默认为1分钟,可以调为10分钟。
hbase.snapshot.region.timeout
hbase.snapshot.master.timeoutMillis