HDDS-15579. Replace SimpleSpanProcessor with BatchSpanProcessor by sravani-revuri · Pull Request #10569 · apache/ozone

sravani-revuri · 2026-06-22T07:00:21Z

What changes were proposed in this pull request?

while running tracing while creating 100000 keys.
with tracing enabled it currently takes around 25-30 min but with it off it takes around 1-2 min.
currently we are using simplespanprocessor which works synchronously for spans being exported , therefore taking so long.

We should change usage to batchspanprocessor which does the same thing asynchronously

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15579

How was this patch tested?

Manual test.

Command:

bash-5.1$ ozone freon rk --num-of-volumes 1 --num-of-buckets 1 --num-of-keys 100000 --num-of-threads 24 --key-size 1KB

BEFORE:

***************************************************
Status: Success
Git Base Revision: 9d50c6884666e794e45102260a4017bb31802e1b
Number of Volumes created: 1
Number of Buckets created: 1
Number of Keys added: 100000
Average Time spent in volume creation: 00:00:00,002
Average Time spent in bucket creation: 00:00:00,000
Average Time spent in key creation: 00:07:39,104
Average Time spent in key write: 00:00:00,298
Total bytes written: 102400000
Total Execution time: 00:36:29,767
***************************************************

AFTER:

***************************************************
Status: Success
Git Base Revision: 9d50c6884666e794e45102260a4017bb31802e1b
Number of Volumes created: 1
Number of Buckets created: 1
Number of Keys added: 100000
Average Time spent in volume creation: 00:00:00,001
Average Time spent in bucket creation: 00:00:00,000
Average Time spent in key creation: 00:00:11,312
Average Time spent in key write: 00:00:00,216
Total bytes written: 102400000
Total Execution time: 00:00:57,661
***************************************************

sravani-revuri · 2026-06-22T07:00:57Z

@sumitagrawl , could you please review these changes.

adoroszlai · 2026-06-23T08:05:07Z

@sumitagrawl please take a look

sarvekshayr

Thanks @sravani-revuri for the improvement.

This reverts commit 9fcff8c.

sumitagrawl

@sravani-revuri it will cause an unbounded accumulation of telemetry data in the heap memory, Please check limiting the memory for this, loosy is fine if crossing it.

jojochuang · 2026-06-26T16:34:02Z

The concern of unbounded heap can be addressed by OTEL_BSP_MAX_QUEUE_SIZE (default=2048). With the default value it's only going to use at most 30MB, which is negligible.

Copilot

Pull request overview

This PR addresses major CLI tracing overhead by switching OpenTelemetry span exporting from synchronous SimpleSpanProcessor to asynchronous BatchSpanProcessor, and by adding a helper to ensure short-lived CLI processes flush buffered spans before exiting.

Changes:

Replace SimpleSpanProcessor with BatchSpanProcessor in TracingUtil and add TracingUtil.execute(...) + flushTracing() for CLI usage.
Update CLI entrypoints (Shell, Freon, OzoneFsShell, OzoneRatis) to use the new TracingUtil.execute(...) wrapper instead of manual init + executeInNewSpan.
Update compose monitoring config to use ozone.tracing.enabled (instead of the deprecated hdds.tracing.enabled).

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/TracingUtil.java	Switch to `BatchSpanProcessor`; add CLI-oriented execute/flush helpers; improve shutdown handling logic.
hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/Shell.java	Use `TracingUtil.execute(...)` to wrap CLI execution in a span with flushing.
hadoop-ozone/freon/src/main/java/org/apache/hadoop/ozone/freon/Freon.java	Wrap Freon execution in `TracingUtil.execute(...)` to avoid sync export overhead.
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneRatis.java	Use `TracingUtil.execute(...)` for tracing span creation + flushing around Ratis shell run.
hadoop-ozone/tools/src/main/java/org/apache/hadoop/fs/ozone/OzoneFsShell.java	Wrap fs command execution in `TracingUtil.execute(...)` and keep conf initialization within the traced scope.
hadoop-ozone/dist/src/main/compose/ozone/monitoring.conf	Switch tracing enablement key to `ozone.tracing.enabled`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sravani-revuri · 2026-06-26T17:41:06Z

+ try {
+ // Best-effort: wait up to 10s for span export; remaining spans may be dropped on exit.
+ batchSpanProcessor.forceFlush().join(10, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Tracing flush: forceFlush failed", e);
+ }


This suggestion was considered not critical according to the above review suggestions. do i go ahead with making the change @jojochuang ?

Copilot doesn't understand prior review comments?

jojochuang

lgtm i'd like to merge along with the first copilot suggestion.

One thing to notice is the thread safety. Variables like sdkTracerProvider should ideally be made a volatile. Same for tracer. That is out of the scope so let's move that to a separate task if needed.

sumitagrawl

LGTM

adoroszlai · 2026-07-06T11:02:41Z

Thanks @sravani-revuri for the patch, @jojochuang, @sarvekshayr, @sumitagrawl for the review.

* master: (519 commits) HDDS-14544. OM DB Insights: Duplicate API calls triggered when changing limit selector (apache#10677). HDDS-15587. [Recon] Show 0 for offline DN pending deletion instead of -1 (apache#10585). HDDS-15521. StreamBlockInputStream fails with TimeoutIOException without retry or failover. (apache#10479) HDDS-15170. Add mock-based unit tests for DataStream write path (apache#10230) HDDS-15552. Ratis events should not be published as metrics (apache#10523) HDDS-15746. Bump kerby to 2.1.2 (apache#10666) HDDS-15579. Replace SimpleSpanProcessor with BatchSpanProcessor (apache#10569) HDDS-15747. Address review comments for HDDS-15083 (apache#10669) HDDS-15732. Some ozone commands ignore OZONE_MODULE_ACCESS_ARGS (apache#10655) HDDS-15605. Fix flaky testContainerExclusionWithClosedContainerException (apache#10621) HDDS-11855. Fix flaky TestContainerBalancerDatanodeNodeLimit#checkIterationResultException (apache#10667) HDDS-15741. Bump awssdk to 2.46.17 (apache#10661) HDDS-10307. Speed up TestOzoneManagerHAWithStoppedNodes (apache#10658) HDDS-15651. Test case for DiskBalancer when markContainerForDelete fails (apache#10593) HDDS-15742. Bump nimbus-jose-jwt to 10.9.1 (apache#10662) HDDS-15719. Add check for allowed action usage in workflows (apache#10641) HDDS-15744. Bump javassist to 3.32.0-GA (apache#10665) HDDS-15737. Fix intermittent failure in balancerShouldOnlySelectConfiguredIncludeContainers (apache#10660) HDDS-15743. Bump gson to 2.14.0 (apache#10664) HDDS-11093. Fix intermittent failure in TestContainerBalancerDatanodeNodeLimit#testMetrics (apache#10659) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScanHelper.java

HDDS-15579. Move from simplespanprocessor to batchspanprocessor.
dbabdf0

adoroszlai reviewed Jun 22, 2026
View reviewed changes

Comment threadhadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/Shell.java Outdated

adoroszlai changed the title ~~HDDS-15579. Move from simplespanprocessor to batchspanprocessor.~~HDDS-15579. Replace SimpleSpanProcessor with BatchSpanProcessorJun 22, 2026

added execute for cli commands
3f7cde0

sravani-revuri requested a review from adoroszlai June 22, 2026 09:23

sravani-revuri marked this pull request as ready for review June 22, 2026 09:24

adoroszlai reviewed Jun 22, 2026
View reviewed changes

Comment threadhadoop-ozone/tools/src/main/java/org/apache/hadoop/fs/ozone/OzoneFsShell.java

fixed errors from tests.
e0a7594

sravani-revuri requested a review from adoroszlai June 22, 2026 16:16

errose28 requested a review from sumitagrawl June 22, 2026 16:21

sarvekshayr reviewed Jun 24, 2026
View reviewed changes

Comment threadhadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/TracingUtil.java

exception handling
9fcff8c

sravani-revuri requested a review from sarvekshayr June 24, 2026 07:35

Revert "exception handling"
7cbad22
This reverts commit 9fcff8c.

adoroszlai requested a review from jojochuang June 24, 2026 16:33

sumitagrawl reviewed Jun 25, 2026
View reviewed changes

adoroszlai mentioned this pull request Jun 26, 2026
HDDS-15552. Ratis events should not be published as metrics #10523
Merged

jojochuang requested a review from Copilot June 26, 2026 16:56

Copilotstarted reviewing on behalf of jojochuang June 26, 2026 16:56 View session

CopilotAI reviewed Jun 26, 2026
View reviewed changes

jojochuang approved these changes Jun 26, 2026
View reviewed changes

Comment threadhadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/TracingUtil.java

jojochuang reviewed Jun 26, 2026
View reviewed changes

Comment threadhadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/tracing/TracingUtil.java Outdated

volatile + shutdown
b7336bf

sumitagrawl approved these changes Jul 6, 2026
View reviewed changes

adoroszlai merged commit be54528 into apache:masterJul 6, 2026
46 of 47 checks passed

Uh oh!

Conversation

sravani-revuri commented Jun 22, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Command:

BEFORE:

AFTER:

Uh oh!

sravani-revuri commented Jun 22, 2026

Uh oh!

Uh oh!

Uh oh!

adoroszlai commented Jun 23, 2026

Uh oh!

sarvekshayr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Jun 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

sravani-revuriJun 26, 2026

Choose a reason for hiding this comment

Uh oh!

adoroszlaiJun 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jojochuang commented Jun 26, 2026•
edited
Loading

adoroszlaiJun 26, 2026•
edited
Loading