HDDS-8101. Add tool to repair broken FSO tree

[sarvekshayr]

What changes were proposed in this pull request?

FSO Repair Tool is introduced to detect and repair broken FSO trees caused by bugs such as HDDS-7592, which can orphan data in the OM.

• This tool identifies the reachable, unreachable (pending for deletion) and unreferenced (dangling) objects .
• Generates a report with the count of directories, files and bytes in each category.
• Snapshots of a given bucket are excluded from repair.
• Allows targeting specific volumes and buckets using --volume and --bucket options.
• In repair mode, the tool deletes only the unreferenced objects.
• By default the tool runs in debug mode to avoid unintentional deletions.

ozone repair om fso-tree --db <dbPath> [--repair | --r] [--volume | -v <volName>] [--bucket | -b <bucketName>] [--verbose]

What is the link to the Apache JIRA

HDDS-8101

How was this patch tested?

Tested the patch manually and added TestFSORepairTool integration test.

Sample output

In debug mode, the tool logs information.

bash-5.1$ ozone repair om fso-tree --db /data/metadata/om.db
ATTENTION: Running as user hadoop. Make sure this is the same user used to run the Ozone process. Are you sure you want to continue (y/N)? y
Run as user: hadoop
FSO Repair Tool is running in debug mode
Creating database of reachable directories at /data/metadata/reachable.db
Processing volume: /s3v
Processing bucket: s3v/fso-bucket
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775031/dir10
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775032/dir9
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775033/dir8
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775034/dir7
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775035/dir6
Reachable:
Directories: 4
Files: 0
Bytes: 0
Unreachable:
Directories: 0
Files: 0
Bytes: 0
Unreferenced:
Directories: 5
Files: 0
Bytes: 0

In repair mode, the unreferenced objects are deleted.

bash-5.1$ ozone repair om fso-tree --db /data/metadata/om.db --repair
ATTENTION: Running as user hadoop. Make sure this is the same user used to run the Ozone process. Are you sure you want to continue (y/N)? y
Run as user: hadoop
FSO Repair Tool is running in repair mode
Creating database of reachable directories at /data/metadata/reachable.db
Processing volume: /s3v
Processing bucket: s3v/fso-bucket
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775031/dir10
Deleting unreferenced directory /-4611686018427388160/-9223372036854775552/-9223372036854775031/dir10
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775032/dir9
Deleting unreferenced directory /-4611686018427388160/-9223372036854775552/-9223372036854775032/dir9
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775033/dir8
Deleting unreferenced directory /-4611686018427388160/-9223372036854775552/-9223372036854775033/dir8
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775034/dir7
Deleting unreferenced directory /-4611686018427388160/-9223372036854775552/-9223372036854775034/dir7
Found unreferenced directory: /-4611686018427388160/-9223372036854775552/-9223372036854775035/dir6
Deleting unreferenced directory /-4611686018427388160/-9223372036854775552/-9223372036854775035/dir6
Reachable:
Directories: 4
Files: 0
Bytes: 0
Unreachable:
Directories: 0
Files: 0
Bytes: 0
Unreferenced:
Directories: 5
Files: 0
Bytes: 0

After repair, the unreferenced objects count is zero.

bash-5.1$ ozone repair om fso-tree --db /data/metadata/om.db
ATTENTION: Running as user hadoop. Make sure this is the same user used to run the Ozone process. Are you sure you want to continue (y/N)? y
Run as user: hadoop
FSO Repair Tool is running in debug mode
Creating database of reachable directories at /data/metadata/reachable.db
Processing volume: /s3v
Processing bucket: s3v/fso-bucket
Reachable:
Directories: 4
Files: 0
Bytes: 0
Unreachable:
Directories: 0
Files: 0
Bytes: 0
Unreferenced:
Directories: 0
Files: 0
Bytes: 0