Skip to content

Generate KnownTags from a tag-conventions registry (phase 2) - #11961

Closed
dougqh wants to merge 1 commit into
dougqh/dense-bloom-filterfrom
dougqh/tag-registry-generator
Closed

Generate KnownTags from a tag-conventions registry (phase 2)#11961
dougqh wants to merge 1 commit into
dougqh/dense-bloom-filterfrom
dougqh/tag-registry-generator

Conversation

@dougqh

Copy link
Copy Markdown
Contributor

Code-generates the KnownTags registry from a language-agnostic tag-conventions.yaml (+ a Java
overlay), replacing the hand-maintained version. Based on dougqh/tag-id-api (the dense-store
branch). Draft — the generator works end to end (main compiles, tests pass); the committed-output
wiring is the remaining piece.

From Claude:

Pipeline (buildSrc task dd-trace-java.tag-registry-generator): parse → resolve per-type tag sets
(extends/include/applies) → assign ids → assign slots by deterministic graph coloring over
per-type layers + an explicit trace_level layer → emit KnownTags.java.

  • Ids encode serial | slot | intercepted (no name hash — known ids are dense-store addressed;
    the name hash is the custom/bucket path). Emitted as literals (javac constant-folds them) with a
    // tagId(...) derivation comment for auditability.
  • slot is a fast-path bitmask bit, not positional storage — co-occurrence is tolerated (the scan
    is authoritative). Coloring gives co-occurring tags distinct bits (fewer scans); colorability
    derives from required. The whole mask fits one long.
  • Trace-level is its own tier (trace_level section → its own TagMap layer); the per-span base
    type holds parent_id/_dd.integration/_dd.svc_src (core-set but per-span).
  • Virtual keys (service/resource.name/error/sampling.priority/…) get reserved ids; the
    generated id → handler dispatch table is the on-ramp to retiring the TagInterceptor.

Verified: internal-api main compiles; KnownTagsTest (property-based resolver round-trip over ~55
tags) passes; OptimizedTagMapDenseForkedTest passes; spotless clean. Note: literal ids fold, so a
constant reference no longer triggers <clinit> — registration is via KnownTags.init().

Remaining (why draft): the generated KnownTags.java is a manual copy in src/main right now;
wire the task to emit to a committed src/generated/java + srcDir + a verifyGenerated freshness
check + format-on-generate, and route the layout reports (folded-types/tag-assignment) there as
committed artifacts.

🤖 Generated with Claude Code

@dougqhdougqh added comp: core Tracer core comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring labels Jul 15, 2026
@dd-octo-sts

dd-octo-stsBot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

SuiteStatus
Startup🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
ScenarioCandidatemasterΔ (95% CI of mean)
startup:insecure-bank:iast:Agent14.12 s14.02 s[-0.3%; +1.7%] (no difference)
startup:insecure-bank:tracing:Agent12.92 s12.99 s[-1.3%; +0.2%] (no difference)
startup:petclinic:appsec:Agent16.52 s16.39 s[-5.8%; +7.3%] (unstable)
startup:petclinic:iast:Agent16.88 s16.88 s[-1.0%; +1.0%] (no difference)
startup:petclinic:profiling:Agent16.71 s16.81 s[-1.9%; +0.7%] (no difference)
startup:petclinic:sca:Agent16.93 s16.78 s[-0.1%; +1.8%] (no difference)
startup:petclinic:tracing:Agent16.10 s15.99 s[-0.3%; +1.7%] (no difference)

Commit:369a591d · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh
dougqh changed the base branch from dougqh/tag-id-api to dougqh/dense-bloom-filterJuly 15, 2026 21:07
@dougqh
dougqhforce-pushed the dougqh/tag-registry-generator branch from 6a83a99 to fdf3c4aCompareJuly 15, 2026 21:07
@dougqh
dougqhforce-pushed the dougqh/dense-bloom-filter branch from 267ebad to 56550cbCompareJuly 15, 2026 21:23
@datadog-prod-us1-3

datadog-prod-us1-3Bot commented Jul 15, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 97.40%
Overall Coverage: 57.31% (-0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 369a591 | Docs | Datadog PR Page | Give us feedback!

@dougqh
dougqhforce-pushed the dougqh/tag-registry-generator branch from fdf3c4a to ad3c1f7CompareJuly 15, 2026 21:25
dougqh added a commit that referenced this pull request Jul 15, 2026
set(long id, Object value) is the id-keyed insertion path: the caller passes a
resolved KnownTags id, so it skips the keyOf name resolution the set(String, ...)
methods pay and stores densely. The id must be a stored known-tag id; custom
names have no id and use the name-keyed setters. The name is resolved lazily only
to clear a read-through tombstone (rare).
Adds TagMapInsertionComparisonBenchmark (dense vs HashMap insertion, -prof gc) —
the isolated micro where the dense store + bloom insertion win shows, and the
home for the id-keyed vs string-keyed comparison.
Reconciled onto the tag registry (#11961).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh
dougqhforce-pushed the dougqh/dense-bloom-filter branch from 56550cb to 75950beCompareJuly 20, 2026 16:51
@dougqhdougqh changed the title Generate KnownTags from a tag-conventions registryGenerate KnownTags from a tag-conventions registry (phase 2)Jul 20, 2026
Replace the hand-written KnownTags with a build-time code generator: a
language-agnostic tag-conventions.yaml (span types via extends/include/applies)
+ a Java overlay (tag-conventions.java.yaml: intercepted hints + the virtual
registry) drive a buildSrc generator that emits KnownTags.java (id constants +
keyOf/nameOf resolver) to committed src/generated. A verifyKnownTags check
regenerates and byte-compares to catch stale commits; src/generated is excluded
from spotless (the emitter produces google-java-format-clean output).
- buildSrc: TagRegistryGeneratorPlugin + generate/verify tasks + parse/resolve
(TagConventions) + id-assignment/coloring (TagRegistry) + emit (KnownTagsEmitter).
- KnownTagCodec gains the name-free 3-arg tagId(globalSerial, intercepted, slot)
the generator uses.
- Generated per-type layout reports (resolved/tag-assignment/layout-by-type/
folded-types) committed as auditable artifacts — the per-type counts are the
substrate the (follow-on) per-type dense-array sizing reads.
- Emitter targets StringIndex.EmbeddingSupport (the keyOf table's open-addressed
static API); the earlier StringIndex.Support name predated its rename.
Substrate only: this lands the generated ids/resolver + the codegen source of
truth; it does NOT yet change dense-array sizing (that per-type-sizing step is
the follow-on that climbs out of the alloc valley). Reconciled onto the dense +
bloom stack; generated ids cohere with the dense store (forked/fuzz green).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dougqh
dougqhforce-pushed the dougqh/tag-registry-generator branch from ad3c1f7 to 369a591CompareJuly 22, 2026 16:32
@datadog-prod-us1-3

datadog-prod-us1-3Bot commented Jul 22, 2026

Copy link
Copy Markdown

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

@dougqh

Copy link
Copy Markdown
ContributorAuthor

Superseded by #12047 — the tag-registry code generator, rebuilt on top of the clean dense #12045 / two-tier bloom #12046 layers. Closing in favor of #12047.

@dougqhdougqh closed this Jul 23, 2026
dougqh added a commit that referenced this pull request Jul 23, 2026
set(long id, Object value) is the id-keyed insertion path: the caller passes a
resolved KnownTags id, so it skips the keyOf name resolution the set(String, ...)
methods pay and stores densely. The id must be a stored known-tag id; custom
names have no id and use the name-keyed setters. The name is resolved lazily only
to clear a read-through tombstone (rare).
Adds TagMapInsertionComparisonBenchmark (dense vs HashMap insertion, -prof gc) —
the isolated micro where the dense store + bloom insertion win shows, and the
home for the id-keyed vs string-keyed comparison.
Reconciled onto the tag registry (#11961).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dougqh added a commit that referenced this pull request Jul 23, 2026
set(long id, Object value) is the id-keyed insertion path: the caller passes a
resolved KnownTags id, so it skips the keyOf name resolution the set(String, ...)
methods pay and stores densely. The id must be a stored known-tag id; custom
names have no id and use the name-keyed setters. The name is resolved lazily only
to clear a read-through tombstone (rare).
Adds TagMapInsertionComparisonBenchmark (dense vs HashMap insertion, -prof gc) —
the isolated micro where the dense store + bloom insertion win shows, and the
home for the id-keyed vs string-keyed comparison.
Reconciled onto the tag registry (#11961).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: coreTracer corecomp: toolingBuild & Toolingtag: ai generatedLargely based on code generated by an AI or LLMtag: no release notesChanges to exclude from release notestype: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@dougqh