本人是hadoop新手,在一次应用中配置出现了问题:就是hadoop 的map阶段正常,但是reduce却卡在18%哪里,一直要将近一个小时才能完成,查看日志如下:
2011-10-03 09:45:58,330 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201110022127_0003_m_000011
2011-10-03 09:46:01,334 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000011_0' has completed task_201110022127_0003_m_000011 successfully.
2011-10-03 09:46:01,334 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:9 completedMapsInputSize:437327225 completedMapsOutputSize:193
2011-10-03 09:46:01,737 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000006_0' has completed task_201110022127_0003_m_000006 successfully.
2011-10-03 09:46:01,737 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:10 completedMapsInputSize:504436090 completedMapsOutputSize:215
2011-10-03 09:46:01,738 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000007_0' has completed task_201110022127_0003_m_000007 successfully.
2011-10-03 09:46:01,738 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:11 completedMapsInputSize:571544955 completedMapsOutputSize:237
2011-10-03 09:46:04,007 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000009_0' has completed task_201110022127_0003_m_000009 successfully.
2011-10-03 09:46:04,007 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:12 completedMapsInputSize:593329451 completedMapsOutputSize:258
2011-10-03 09:46:04,008 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000008_0' has completed task_201110022127_0003_m_000008 successfully.
2011-10-03 09:46:04,008 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:13 completedMapsInputSize:615113993 completedMapsOutputSize:279
2011-10-03 09:46:13,349 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for task attempt_201110022127_0003_m_000000_0
2011-10-03 09:48:49,450 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #2 for task attempt_201110022127_0003_m_000000_0
2011-10-03 09:53:52,659 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #3 for task attempt_201110022127_0003_m_000000_0
2011-10-03 09:53:52,659 INFO org.apache.hadoop.mapred.JobInProgress: Too many fetch-failures for output of task: attempt_201110022127_0003_m_000000_0 ... killing it
2011-10-03 09:53:52,659 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201110022127_0003_m_000000_0: Too many fetch-failures
2011-10-03 09:53:52,661 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201110022127_0003_m_000000_1' to tip task_201110022127_0003_m_000000, for tracker 'tracker_ubuntu-server:127.0.0.1/127.0.0.1:49740'
2011-10-03 09:53:52,661 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201110022127_0003_m_000000
2011-10-03 09:53:53,107 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201110022127_0003_m_000000_0' from 'tracker_fangfei-desktop:127.0.0.1/127.0.0.1:54181'
2011-10-03 09:53:58,264 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201110022127_0003_m_000000_2' to tip task_201110022127_0003_m_000000, for tracker 'tracker_huangzhongyuan-desktop:127.0.0.1/127.0.0.1:48184'
2011-10-03 09:53:58,264 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201110022127_0003_m_000000
2011-10-03 09:54:01,668 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201110022127_0003_m_000000_1' has completed task_201110022127_0003_m_000000 successfully.
2011-10-03 09:54:01,668 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:14 completedMapsInputSize:682222858 completedMapsOutputSize:301
2011-10-03 09:54:07,282 INFO org.apache.hadoop.mapred.JobTracker: Adding task (cleanup)'attempt_201110022127_0003_m_000000_2' to tip task_201110022127_0003_m_000000, for tracker 'tracker_huangzhongyuan-desktop:127.0.0.1/127.0.0.1:48184'
2011-10-03 09:54:10,285 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201110022127_0003_m_000000_2' from 'tracker_huangzhongyuan
在网上google了很久没找到答案,后来有的说把/etc/hosts中的127.0.0.1的主机映射去掉,试了也不行。知道第二天上午google 的时候才受到一点启发,突然想到,这个问题是发生在reduce阶段,而提示的消息应该是取不到map阶段的结果,既然在Failed fetch notification #1 for task attempt_201110022127_0003_m_000000_0 中有取不到的任务分块的名字,说明namenode正常工作,namenode通知reduce节点进行reduce操作,而它却取不到,只能说明它没法 和那些节点通信,又由于我在配置hadoop的时候用的是主机的名字,不是ip,所以想到解决办法应该是把各个datanode节点的映射互相加到 /etc/hosts中。试了一下,果然正确。所以在此记录。
解决办法 :把各个datanode节点的映射都加到每个datanode节点的/etc/hosts中