在hdfs上创建了一个文件
[hadoop@hdp02 ~]$ echo "hadoop file" > hadoop01.txt
[hadoop@hdp02 ~]$ hadoop fs -mkdir /hadoop
[hadoop@hdp02 ~]$ hadoop fs -put hadoop01.txt /hadoop/
[hadoop@hdp02 ~]$ hadoop fs -ls -h /hadoop
Found 1 items
-rw-r--r-- 3 hadoop supergroup 19 2020-04-05 21:02 /hadoop/hadoop01.txt
使用fsck命令进行hdfs健康检查,发现了文件异常,配置副本为3,只有2副本
#hdfs文件健康检查
[hadoop@hdp02 ~]$ hdfs fsck /
Connecting to namenode via http://hdp01:50070/fsck?ugi=hadoop&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.60.101 for path / at Sun Apr 05 22:53:38 CST 2020
.
/hadoop/hadoop01.txt: Under replicated BP-417206835-192.168.60.101-1586085282109:blk_1073741825_1001. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
Status: HEALTHY
Total size: 12 B
Total dirs: 2
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 12 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Sun Apr 05 22:53:38 CST 2020 in 1 milliseconds
The filesystem under path '/' is indeed HEALTHY
异常信息显示,一个hdfs 文件块丢失
执行hdfs文件手动恢复命令
hdfs debug recoverLease -path /hadoop/hadoop01.txt -retries 10
然后再查看健康状况,执行hdfs fsck,发现信息还是和上面一样。问题出在哪?
后来想想,我的集群只配置了一台namenode和2台datanode,文件副本只会存储在datanode节点上。所以就算我的hdfs-site.xml文件配置了副本数为3,实际也最多只能存在2个副本!
没办法,需要增加一个datanode节点。编辑 $HADOOP_HOME/etc/hadoop/slaves文件,增加hdp01为新的datanode节点,然后将该文件同步到其他节点。
增加了datanode节点,集群性质发生了变化,需要重新初始化namenode,格式化需要进行一系列步骤,否则格式化无效!
如何格式初始化namenode,参见我的 hdfs datanode进程没法启动
完成格式化后,创建新的文件
再次执行hdfs fsck / 查看hdfs 文件系统状况
[hadoop@hdp02 hadoop]$ hadoop fs -put README.txt /hadoop/
[hadoop@hdp02 hadoop]$ hdfs fsck /
Connecting to namenode via http://hdp01:50070/fsck?ugi=hadoop&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.60.102 for path / at Mon Apr 06 00:01:24 CST 2020
.Status: HEALTHY
Total size: 1366 B
Total dirs: 2
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 1366 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Apr 06 00:01:24 CST 2020 in 5 milliseconds
The filesystem under path '/' is HEALTHY