HDDS-15496. SCM async container ID export CLI

[sarvekshayr]

What changes were proposed in this pull request?

Adds ozone debug scm container export to run an async export on the SCM leader. Matching container IDs (by lifecycle and/or health state) are written as newline-delimited text shards and packed into a TAR under {scm.db.dirs}/exports/. Use export status --job-id to check progress; completed jobs report a fixed elapsedMs and the TAR path. Delete export files manually when no longer needed.

What is the link to the Apache JIRA

HDDS-15496

How was this patch tested?

Sample output

# ozone debug scm container export --health-state MISSING
Submitted container ID export job: 8520cf61-05c5-4934-976c-3ebcd36649b0
Export directory on SCM leader: /var/lib/hadoop-ozone/scm/data/exports
Check status with: ozone debug scm container export status --job-id 8520cf61-05c5-4934-976c-3ebcd36649b0
# ozone debug scm container export status --job-id 8520cf61-05c5-4934-976c-3ebcd36649b0
jobId=8520cf61-05c5-4934-976c-3ebcd36649b0
state=SUCCEEDED
totalRows=41,851
elapsedMs=226
exportDir=/var/lib/hadoop-ozone/scm/data/exports
healthState=MISSING
tarPath=/var/lib/hadoop-ozone/scm/data/exports/container-ids-health-MISSING-20260706T064125Z-8520cf61-05c5-4934-976c-3ebcd36649b0.tar
Delete the TAR file on the SCM leader manually when no longer needed.
# cat container-ids-health-MISSING-20260706T054532Z-6e6891cb-2b86-4964-a36c-0d444b994ca3.tar
container-ids-health-MISSING-2026-07-06T05:45:32Z-part001.txt0100644 0000000 0000000 00001050561 15222640374 021044 0ustar000000000 0000000 # jobId=6e6891cb-2b86-4964-a36c-0d444b994ca3
# timestamp=2026-07-06T05:45:32Z
# healthState=MISSING
# startContainerId=1013
# part=1
# format=container-id-per-line
1013
1016
1034
1054
1055
1075
1088
1102
1103
1120
1136
1137
...

Performance analysis

Cluster details

Total containers: 473,641
CLOSED containers: 436,500
Test date: Jul 8, 2026 (~04:51–05:01 UTC)

Test environment note -
Concurrent load during all export runs: Containers were being created simultaneously in the background while the export CLI was running.
Key size: 1 MB
Container size: 1 MB

First runs (cold JVM / after idle)

No	Filter	Job ID (prefix)	Rows	Time
1	CLOSED	60176f71…	436,500	11,871 ms
2	MISSING	c09583da…	41,851	1,922 ms
3	MISSING + CLOSED	5b7a8a8d…	41,843	1,311 ms

Subsequent runs (warm JVM)

No	Filter	Job ID (prefix)	Rows	Time
4	CLOSED	bd47f466…	436,539	280 ms
5	MISSING	70115c06…	41,851	63 ms
6	MISSING + CLOSED	0fdbd67a…	41,843	53 ms

First run (--lifecycle-state CLOSED)

~0 → 330k rows in ~6.2 s (~53k IDs/s) — steady
330k → 352k rows in ~5.0 s (~4k IDs/s) — major slowdown
352k → 436.5k rows in ~0.6 s — fast again
JVM / GC (from Prometheus screenshot)

Large GC spike during 04:51–04:53 UTC: ~4,000–5,000 ms GC time in that window
Aligns with the mid-run stall (~330k–352k rows)
Conclusion: First run was slow mainly due to cold JVM + heavy GC, not slow lifecycle indexing alone

[jojochuang]

cc @sreejasahithi you might be interested in this one

[Copilot]

Pull request overview

This PR adds an asynchronous “container ID export” capability to SCM, exposed via a new ozone debug scm container export CLI. It introduces server-side job management that writes matching container IDs into newline-delimited shard files, packs them into a TAR on the SCM leader, and provides an RPC/CLI status endpoint to poll job progress and retrieve the TAR path.

Changes:

• Added SCM server support for async container ID export jobs, including status reporting and audit logging.
• Extended container-ID listing APIs to support filtering by container health state (in addition to lifecycle state), and updated Recon SCM sync call sites accordingly.
• Added a TAR append utility in Archiver plus tests, and wired new CLI debug commands to submit/poll export jobs.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconStorageContainerSyncHelper.java	Updates mocks for new getListOfContainerIDs(..., healthState) signature.
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconSCMContainerSyncIntegration.java	Updates integration test mocks for health-state parameter.
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java	Extends SPI to accept an optional container health-state filter.
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java	Passes health-state filter through to SCM client.
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerSyncHelper.java	Updates Recon SCM sync to call new getListOfContainerIDs(..., null) signature.
hadoop-ozone/dist/src/main/compose/ozone/docker-config	Adds a config entry for SCM export directory in compose environment.
hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/scm/SCMDebug.java	Introduces scm debug subcommand root.
hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/scm/package-info.java	Package documentation for new SCM debug command namespace.
hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/scm/container/package-info.java	Package documentation for container debug commands.
hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/scm/container/ContainerSubCommand.java	Adds container debug subcommand and wires export.
hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/scm/container/ExportContainerIDs.java	Implements ozone debug scm container export submission + optional watch loop.
hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/scm/container/ExportContainerStatus.java	Implements export status --job-id output formatting.
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java	Adds client methods to submit export jobs and fetch export status via SCM RPC.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManager.java	Updates state-manager tests for new health-state parameter.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/export/TestContainerExportManager.java	Adds unit tests for export manager (sharding, TAR creation, job behavior).
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/ozone/audit/SCMAction.java	Adds audit action constant for container ID export.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java	Instantiates and shuts down the new ContainerExportManager.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java	Adds RPC endpoints for submit/status and extends ListContainerIDs with health filter.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java	Adds protobuf translation for submit/status RPCs and healthState parsing for ListContainerIDs.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/invoker/ContainerStateManagerInvoker.java	Updates HA invoker method signature for health-state-aware container ID retrieval.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/states/ContainerStateMap.java	Adds health-state filtering support when listing container IDs.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/package-info.java	Package documentation for SCM container export implementation.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/export/ContainerExportManager.java	Implements async export job execution, sharding, TAR writing, and status reporting.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManagerImpl.java	Plumbs health-state parameter into state-manager list operation.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java	Extends interface to accept optional health-state filter.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java	Extends container manager list method to pass through health state.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManager.java	Updates container manager API to accept health-state filter.
hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto	Adds new RPC protos for submit/status and adds healthState to ListContainerIDs request.
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestArchiver.java	Adds test coverage for TAR append behavior.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/Archiver.java	Adds appendFile and EOF-stripping logic to append to TARs.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java	Adds client-side translation for submit/status RPCs and healthState in ListContainerIDs.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java	Extends protocol with submit/status export APIs and health-state filter param.
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java	Extends client interface with submit/status export APIs.
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/export/package-info.java	Package documentation for shared export status types.
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/export/ContainerExportStatus.java	Adds client-visible export job status DTO.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ArafatKhan2198]

Thanks for your effort on this @sarvekshayr
This will be a very useful tool!

Some comments for you -

Before merge I'd prioritize:
(1) fixing the leader/HA story or the misleading "leader" wording
(2) bounding jobTracker
(3) adding unit tests for the new ContainerStateMap filtering paths.

[devmadhuu]

Thanks @sarvekshayr , few comments pls check.

[devmadhuu]

This job status is being fetched based on checkAdminAccess called inside SCMClientProtocolServer.submitContainerIdExport/getContainerIdExportStatus , so what if leader changes after submitting job, the new leader has no record → export status throws "Export job not found", while the original job keeps running on the now-follower and drops a TAR on a node the operator wasn't told about.

[sarvekshayr]

Export runs only on the SCM leader, with job status kept in memory on that node. If leadership changes or SCM restarts, the in-flight job stops, partial artifacts are cleaned up, and the job ID is no longer queryable on the new leader. Operator must re-submit on the current leader. Completed TAR files remain on disk (under ozone.scm.container.export.dir, default {scm.db.dirs}/exports).

[szetszwo]

@sarvekshayr , this PR is very big. Could you split it into a few PRs? The first one could make the change in SCM.

[sarvekshayr]

@sarvekshayr , this PR is very big. Could you split it into a few PRs? The first one could make the change in SCM.

Thanks for the suggestion. I'd prefer to keep this as a single PR.

Even with split, SCM half would still touch 20+ files because of the client/server wiring (proto, protocol, translators, config, tests). The feature is already tested end-to-end as one change, and splitting would mean landing incomplete changes across multiple PRs without much reduction in review scope.

[szetszwo]

@sarvekshayr , Software engineering: a big change vs multiple small changes:

[devmadhuu]

Thanks @sarvekshayr for improving the patch. Changes LGTM +1.

[szetszwo]

-1 for the change being too big.

Even with split, SCM half would still touch 20+ files ...

If it is the case, we should further split the change.

[sarvekshayr]

Closing this now. As discussed, I am breaking this work down into smaller PRs.

Tracking Jira - HDDS-15496