Skip to content

Encode tag ids with a colored slot and collapse dense presence to one occupancy long (phase 2) - #12046

Draft
dougqh wants to merge 1 commit into
dougqh/dense-store-v2from
dougqh/bloom-v2
Draft

Encode tag ids with a colored slot and collapse dense presence to one occupancy long (phase 2)#12046
dougqh wants to merge 1 commit into
dougqh/dense-store-v2from
dougqh/bloom-v2

Conversation

@dougqh

@dougqhdougqh commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Reframe of the dense-store presence layer (was: two-tier presence bloom).

Replaces the earlier two-tier design (per-map group-decl mask + field-decl bloom) with a single global colored slot: the tag-id middle 16 bits carry one graph-colored slot coordinate (SLOT_SHIFT=32, SLOT_MASK=0xFFFF), so the dense store tracks presence with one occupancy long instead of a group mask plus a field bloom. Also adds the trace-level bit (LEVEL_TRACE) and level-bit read-through in the parent-visibility check.

KnownTags stays hand-maintained here (src/main); the tag-registry code generator that produces these colored ids lands in #12047, which relocates the file to src/generated.

Stacked on #12045 (dense store). Draft.

🤖 Generated with Claude Code

@dougqhdougqh added comp: core Tracer core tag: no release notes Changes to exclude from release notes type: refactoring tag: ai generated Largely based on code generated by an AI or LLM labels Jul 23, 2026
@datadog-prod-us1-6

datadog-prod-us1-6Bot commented Jul 23, 2026

Copy link
Copy Markdown

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

Wrapped the dense-tag initialization comment in dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java to match the repository formatter.

Commit fix to this PR


View in Datadog | Reviewed commit d6e7088 · Any feedback? Reach out in #deveng-pr-agent

@datadog-prod-us1-6

This comment has been minimized.

@dougqhdougqh changed the title Add bloom-filter fast-path to the dense tag storeAdd a two-tier presence bloom to the dense tag storeJul 23, 2026
@dd-octo-sts

dd-octo-stsBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

SuiteStatus
Startup🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
ScenarioCandidatemasterΔ (95% CI of mean)
startup:insecure-bank:iast:Agent13.93 s13.86 s[-0.2%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent12.90 s12.95 s[-1.1%; +0.4%] (no difference)
startup:petclinic:appsec:Agent16.92 s17.30 s[-6.7%; +2.3%] (no difference)
startup:petclinic:iast:Agent17.35 s17.07 s[-2.7%; +6.0%] (no difference)
startup:petclinic:profiling:Agent17.32 s17.16 s[-0.4%; +2.2%] (no difference)
startup:petclinic:sca:Agent17.45 s17.45 s[-1.0%; +1.0%] (no difference)
startup:petclinic:tracing:Agent16.48 s16.63 s[-1.9%; +0.1%] (no difference)

Commit:bb9ebca1 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqhdougqh changed the title Add a two-tier presence bloom to the dense tag storeAdd a two-tier presence bloom to the dense tag store (phase 2)Jul 23, 2026
@dougqhdougqh changed the title Add a two-tier presence bloom to the dense tag store (phase 2)Encode tag ids with a colored slot and collapse dense presence to one occupancy long (phase 2)Aug 17, 2026
Reframe of the dense-store presence layer. Replaces the earlier two-tier
(group-decl mask + field-decl bloom) design with a single global colored
slot: the tag-id middle 16 bits carry one graph-colored slot coordinate
(SLOT_SHIFT=32, SLOT_MASK=0xFFFF), so the dense store tracks presence with
one occupancy long instead of a group mask plus a field bloom. Adds the
trace-level bit (LEVEL_TRACE) and level-bit read-through in the parent
visibility check.
KnownTags remains hand-maintained here (src/main); the tag-registry code
generator that produces these colored ids lands in the following commit,
which relocates the file to src/generated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh
dougqhforce-pushed the dougqh/dense-store-v2 branch from d758efe to 992dc14CompareAugust 18, 2026 20:14
@dougqh

Copy link
Copy Markdown
ContributorAuthor

@codex review

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:bb9ebca1a0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

// makeTagId(serial=285, slot=13) <conditional>

public static final String HTTP_STATUS_CODE_NAME = "http.status_code";
public static final long HTTP_STATUS_CODE_ID = 0x011E000A00000000L;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mark HTTP status IDs as intercepted

When an ID-based tag caller uses the advertised sign-bit routing, keyOf("http.status_code") now returns this positive ID, so isIntercepted is false even though TagInterceptor.needsIntercept explicitly handles this tag to populate DDSpanContext.httpStatusCode and apply the 404 resource-name rule. Encode this ID with INTERCEPTED, as is done for the other stored tags that require interceptor side effects.

Useful? React with 👍 / 👎.

Comment on lines +43 to +47
public static final String MEASURED_NAME = "measured";
public static final long MEASURED_ID = 0x8009FFFF00000000L;
// makeTagId(serial=9, slot=NO_SLOT) + intercepted [directive]

public static final String ANALYTICS_SAMPLE_RATE_NAME = "analytics.sample_rate";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Register the actual reserved tag names

The new reserved mappings use conceptual names rather than the keys handled by TagInterceptor: for example, the runtime keys are _dd.measured and _dd1.sr.eausr, so keyOf(DDTags.MEASURED) and keyOf(DDTags.ANALYTICS_SAMPLE_RATE) both return zero instead of these reserved IDs. The same mismatch affects _dd.origin (registered as origin) and service.name (only service is registered), preventing established tag constants from using the advertised name-to-ID/interceptor route.

Useful? React with 👍 / 👎.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: coreTracer coretag: ai generatedLargely based on code generated by an AI or LLMtag: no release notesChanges to exclude from release notestype: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@dougqh