动态分区问题,如果数据量大或者当动态分区大甚至只有十几个时,会出现如下异常:
2015-10-23 16:43:54,165 INFO [fetcher#10] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: assigned 20 of 34 to spark-03:13562 to fetcher#10 2015-10-23 16:43:54,166 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#9 2015-10-23 16:43:54,167 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#9 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:304) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:294) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
参考issue:
https://issues.apache.org/jira/browse/MAPREDUCE-6108
https://issues.apache.org/jira/browse/MAPREDUCE-6447
https://issues.apache.org/jira/browse/MAPREDUCE-6447
参数理解:
mapreduce.map.java.opts -xmx配置的 heap memory cloudera mapreduce.map.java.opts.max.heap 一般设置java.opts为memory.mb的75% mapreduce.reduce.java.opts -xmx配置的 heap memory cloudera mapreduce.reduce.java.opts.max.heap 一般设置java.opts为memory.mb的75% mapreduce.map.memory.mb 1G默认 mapreduce.reduce.memory.mb 1G默认 mapreduce.reduce.memory.totalbytes
mapreduce.reduce.shuffle.parallelcopies shuffle开启的fetcher线程数 apache默认5,choudera默认10
mapreduce.reduce.shuffle.input.buffer.percent 默认0.7
mapreduce.reduce.shuffle.memory.limit.percent默认0.25
如上3个参数相乘得小于1,否则将报如上错。
将mapreduce.reduce.shuffle.parallelcopies调成5,可以解决此问题。
另外cloudera hive hive.stats.autogather默认为true,即插入数据时会优化统计,如此在大的动态分区时load数据后会有一段很长时间的统计,且操作hive元数据表,例如每个分区的文件数,行数等等。耗时比较长时可能会timeout,需要将其设成false。