check the logs or run fsck in order to identify the missing blocks

Posted by tomfmason on Wed, 25 Mar 2020 15:31:06 +0100

hadoop version is 2.8.3

Today, I found a strange problem, as shown in List-1 below, indicating that two file blocks are missing

    List-1

There are 2 missing blocks. The following files may be corrupted:

blk_1073857294	/tmp/xxx/b9a11fe8-306a-42cc-b49f-2a7f098ecb5a/hive-exec-2.1.1.jar
blk_1073857295	/tmp/xxx/b9a11fe8-306a-42cc-b49f-2a7f098ecb5a/hive-
hcatalog-core-3.0.0.jar

Please check the logs or run fsck in order to identify the missing blocks. See the Hadoop FAQ for common causes and potential solutions.

Because it is / tmp directory, which is not normal business data, we delete it directly, as shown in List-2 below, and then go to see the hdfs page. There is no such problem.

    List-2

[xx@xxx hadoop]# hadoop  fsck -delete
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Connecting to namenode via http://xxxx:50070/fsck?ugi=root&delete=1&path=%2F
FSCK started by root (auth:SIMPLE) from /10.42.5.26 for path / at Wed Mar 25 12:35:39 CST 2020
..............................................................................
/tmp/xxx/b9a11fe8-306a-42cc-b49f-2a7f098ecb5a/hive-exec-2.1.1.jar: CORRUPT blockpool BP-604784226-10.42.1.102-1577681916881 block blk_1073857294

/tmp/xxx/b9a11fe8-306a-42cc-b49f-2a7f098ecb5a/hive-exec-2.1.1.jar: MISSING 1 blocks of total size 32441258 B..
/tmp/xxx/b9a11fe8-306a-42cc-b49f-2a7f098ecb5a/hive-hcatalog-core-3.0.0.jar: CORRUPT blockpool BP-604784226-10.42.1.102-1577681916881 block blk_1073857295

/tmp/xxx/b9a11fe8-306a-42cc-b49f-2a7f098ecb5a/hive-hcatalog-core-3.0.0.jar: MISSING 1 blocks of total size 269009 B......................
...

Cause analysis:

The data is stored on hdfs in blk_andblk_ways. After blk_andblk_aredeleted, the metadata is still there, but the data block is not there, so this error is reported. But I don't need this part of data, so I can directly delete the metadata information of the abnormal file block.

Reference

1.https://blog.csdn.net/lsr40/article/details/79426333

Topics: Programming hive Hadoop