Generate KnownTags from tag-conventions via the tag-registry code generator (phase 2) by dougqh · Pull Request #12047 · DataDog/dd-trace-java

dougqh · 2026-07-23T01:09:38Z

What Does This Do

Replaces the hand-maintained known-tag table with one generated from a single source of truth (tag-conventions.yaml + a Java overlay), so tag ids / the id↔name resolver / (later) per-type layouts stop drifting across core, instrumentation, and language tracers.

Generator (buildSrc)

Modeled on SupportedConfigPlugin (data file → .java via PrintWriter):

TagRegistryGeneratorPlugin + GenerateKnownTagsTask — read tag-conventions.yaml (domain) + tag-conventions.java.yaml (Java overlay: intercepted hints + virtual/special registry), resolve per-type tag sets (extends/include/applies, de-duped), assign ids in stable serial order, and greedily graph-color the colorable set into slots.
KnownTagsEmitter — emits google-java-format-clean Java to committed internal-api/src/generated with a literal id + // tagId(serial=…, group=…, field=…) audit comment (literal ⟹ javac constant-folds; comment ⟹ auditable).
verifyKnownTags — regenerates + git-diffs to catch stale commits. Generated classes are jacoco-excluded.

Committing the generated sources also lets us audit what ids/layout any release branch shipped via git checkout.

internal-api

Deletes the interim hand-maintained src/main/.../KnownTags.java in favor of the generated src/generated/.../KnownTags.java (byte-equal ids — the equivalence gate).
Read-through stays chain-aware: keeps this stack's multi-level parentDenseVisible over the pre-reconcile single-level form.
Generated keyOf table uses StringIndex.EmbeddingSupport (matching the tagset base rename) in both the emitter and its committed output, so regeneration is drift-free.

Verification

:internal-api:generateKnownTags (no drift), verifyKnownTags, KnownTagsTest, TagMapDenseForkedTest, TagMapDenseFuzzForkedTest, spotlessJavaCheck, spotbugsMain — all green.

Stack

tagset → dense-store (#12045) → two-tier bloom (#12046) → **generator**. Base dougqh/bloom-v2. Supersedes old #11961.

Non-goals (follow-on)

Wiring SpanLayouts into the dense store; full virtual-dispatch table → TagInterceptor retirement; bandwidth-minimizing packer (greedy is v1).

🤖 Generated with Claude Code

datadog-prod-us1-4 · 2026-07-23T01:14:35Z

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

CoreTracer.java had a line exceeding the required formatter’s wrapping rules. Wrapped the dense-tag initialization comment to comply with the repository style.

Commit fix to this PR

View in Datadog | Reviewed commit e6800b8 · Any feedback? Reach out in #deveng-pr-agent

datadog-prod-us1-4 · 2026-07-23T01:26:56Z

🎯 Code Coverage (details)
• Patch Coverage: 86.11%
• Overall Coverage: 57.39% (+0.01%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 33f32c4 | Docs | View more details | Give us feedback!}

dd-octo-sts · 2026-07-23T01:33:24Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.76 s	14.69 s	[-0.3%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	13.59 s	13.71 s	[-1.8%; +0.0%] (no difference)
startup:petclinic:appsec:Agent	16.89 s	16.76 s	[-0.3%; +1.8%] (no difference)
startup:petclinic:iast:Agent	16.92 s	16.95 s	[-0.9%; +0.6%] (no difference)
startup:petclinic:profiling:Agent	16.61 s	16.74 s	[-1.9%; +0.3%] (no difference)
startup:petclinic:sca:Agent	16.82 s	16.61 s	[+0.2%; +2.4%] (maybe worse)
startup:petclinic:tracing:Agent	16.07 s	16.04 s	[-0.7%; +1.0%] (no difference)

Commit:33f32c45 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Reconciles #11901's SizingHint half onto the colored generator-v2 base as its own layer. A per-operation SizingHint (kept in a bounded, self-tuning SizingHintTable keyed by operation name, two lanes for entry vs child spans) sizes a span's dense TagMap at create and records the observed known-tag high-water mark back on finish, so the reused hint converges to the operation's real size. Erases the fixed KNOWN_INIT_CAP dense-array floor tax that regressed bare/small spans in the #12047-vs-1.65 A/B; the id-keyed write API stacks on top of this as a separate layer. New: SizingHint, SizingHelper, SizingHintTable, FlatHashtable (+ tests, jmh). TagMap: create(SizingHint), recordSize, denseCapHint. DDSpanContext threads a sizingHint through the primary ctor + recordDenseSize on finish; DDSpan hooks it at finishAndAddToTrace; CoreTracer resolves the per-operation hint/lane. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dougqh · 2026-08-18T14:12:52Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:098070fd3e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-18T14:21:41Z

+ this.traceDenseTagsEnabled =
+ configProvider.getBoolean(TracerConfig.TRACE_DENSE_TAGS_ENABLED, false);


Log the new dense-tags configuration
The new configuration is stored and exposed but never added to Config.toString(). Consequently, startup/configuration diagnostics omit whether this experimental path is enabled, making deployment and experiment failures unnecessarily difficult to diagnose; the repository's configuration procedure explicitly requires every new configuration to be included there.
AGENTS.md reference: AGENTS.md:L37-L43
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-18T14:21:42Z

+ val outDir = destinationDirectory.get().asFile
+ TagRegistryGenerator.generate(domainYaml.get().asFile, overlayYaml.get().asFile, outDir)


Remove obsolete outputs before regenerating
When a later generator revision renames or stops emitting a report/source file, generateKnownTags only overwrites current outputs and leaves the retired file in src/generated. verifyKnownTags then reports that file as stale while instructing developers to rerun this task, but rerunning cannot resolve the failure; clear the owned destination tree before generating so it represents the exact current output set.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-18T14:21:42Z

+ Tag(
+ name = m["tag"].toString(),
+ type = (m["type"] as? String) ?: "string",
+ required = (m["required"] as? String) ?: "optional",
+ )


Preserve aliases when parsing tag declarations
When dense tags are enabled and instrumentation uses one of the aliases explicitly declared in the new source of truth—such as http.request.method, http.response.status_code, url.full, or db.system—the parser discards aliases, so KnownTags.RESOLVER.keyOf(...) treats that key as unknown and leaves it on the bucket path instead of associating it with the declared known tag. Parse and emit alias-to-ID entries so the generated registry actually covers the semantic aliases represented by the YAML.
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-18T14:21:42Z

 // ignore embedded test projects and everything in build dir, e.g. generated sources
- targetExclude('src/test/resources/**', buildDirectoryFiles)
+ // src/generated/** is emitted by code generators (e.g. the tag registry) — verified by their own freshness gate
+ targetExclude('src/test/resources/**', 'src/generated/**', buildDirectoryFiles)


Keep generated Java subject to the formatting standard
Excluding every src/generated/** Java source removes the repository's mandatory Google Java Format check from committed generated code, and the newly committed KnownTags.java already contains an assignment that the formatter wraps differently. This lets future emitter changes accumulate nonconforming Java undetected; make the emitter produce formatter-clean output or format generated Java before freshness verification rather than globally bypassing Spotless.
AGENTS.md reference: AGENTS.md:L56-L59
Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-18T14:21:42Z

+ a.appendLine("# STORED serial slot int lvl id required name")
+ for (t in reg.stored) {
+ a.appendLine(
+ " %6d %5s %s %s %-18s %-12s %s".format(


Make generated reports independent of the default locale
On machines whose JVM default locale uses non-ASCII decimal digits, the unqualified String.format calls localize %d output, so the generated report bytes differ from the committed files even with identical YAML inputs. This contradicts the generator's byte-identical-output invariant and makes verifyKnownTags fail solely because of the developer or CI locale; format all generated output with Locale.ROOT.
Useful? React with 👍 / 👎.

Reconciles #11901's SizingHint half onto the colored generator-v2 base as its own layer. A per-operation SizingHint (kept in a bounded, self-tuning SizingHintTable keyed by operation name, two lanes for entry vs child spans) sizes a span's dense TagMap at create and records the observed known-tag high-water mark back on finish, so the reused hint converges to the operation's real size. Erases the fixed KNOWN_INIT_CAP dense-array floor tax that regressed bare/small spans in the #12047-vs-1.65 A/B; the id-keyed write API stacks on top of this as a separate layer. New: SizingHint, SizingHelper, SizingHintTable, FlatHashtable (+ tests, jmh). TagMap: create(SizingHint), recordSize, denseCapHint. DDSpanContext threads a sizingHint through the primary ctor + recordDenseSize on finish; DDSpan hooks it at finishAndAddToTrace; CoreTracer resolves the per-operation hint/lane. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Introduce the tag-registry code generator (buildSrc plugin + tag-conventions YAML) and use it to assign the colored slots from the previous commit by graph-coloring the tag co-occurrence graph (each concrete span type's resolved set plus the <trace> clique), packing all stored tags into SLOT_COUNT=16 slots. Relocates the hand-maintained KnownTags from src/main to the generated src/generated source set and renames the codec accessors to their generated form (serialNum/makeTagId). Adds the generator verify task so CI fails if the checked-in KnownTags drifts from the conventions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Expose the known-tag id on EntryReader: TagMap.Entry resolves it lazily via KnownTagCodec.keyOf (same memoized-field idiom as lazyTagHash; 0L means unknown tag / inactive codec, so the sentinel is Long.MIN_VALUE). The dense reader flyweight already has the id in hand at emit time and records it directly, skipping the keyOf resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ting Address two Codex robustness findings on the generator: - generateKnownTags now clears its owned destination tree before regenerating, so a report/source file retired by a later revision cannot linger and fail verifyKnownTags with no way for the fix task to remove it. - All String.format calls in the report/emitter use Locale.ROOT, so %d output no longer localizes on machines whose default locale uses non-ASCII digits, preserving the byte-identical-output invariant verifyKnownTags relies on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Address the Codex finding that the new experimental dense-tags flag was stored but never surfaced in Config.toString (repo convention requires new configs appear there for diagnostics). Surface it only when it diverges from the default, so normal config dumps stay uncluttered, and compare against a new DEFAULT_TRACE_DENSE_TAGS_ENABLED constant (also now the getBoolean default) so the condition stays correct if the default ever changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Address the Codex finding that gradle/spotless.gradle excluded src/generated/**, bypassing google-java-format on committed generated code. Drop the exclude and make KnownTagsEmitter emit google-java-format-clean output (blank line before the keyOf static initializer; wrap the long id-assignment), so spotlessCheck and the byte-identical verifyKnownTags gate both pass on the emitted file. internal-api is the only module with a committed src/generated Java tree, so the exclude removal is otherwise a no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Adds a per-namespace name to the tag registry: each tag may declare an `open-telemetry-name` (replacing the vestigial `aliases` list). The generator now: - parses it off both YAMLs into the model, - validates it (an OTel name may not collide with a canonical tag name nor be claimed by two tags -> build fails loudly rather than silently picking one), - emits it into the keyOf table so keyOf(otelName) resolves to the canonical tag's id (inbound, many->one), and - emits a reverse switch so openTelemetryNameOf(tagId) recovers it (outbound). KnownTagCodec gains datadogTagOf(tagId) (== nameOf, the canonical name) and openTelemetryTagOf(tagId) (the OTel name, or null). nameOf is unchanged and still returns the Datadog name -- outbound is namespace-specific, not normalized. Serializer applicability (when a span renders under OTel names, fallback policy) and additional namespaces are a follow-on concern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dougqh · 2026-08-18T21:26:17Z

@codex review

chatgpt-codex-connector · 2026-08-18T21:29:44Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit:33f32c450a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

This was referenced Jul 23, 2026
Store known span tags densely in TagMap by tag-id (phase 2) #11814
Closed
Generate KnownTags from a tag-conventions registry (phase 2) #11961
Closed

dougqh added comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring labels Jul 23, 2026

dougqh commented Jul 23, 2026
View reviewed changes

Comment threadinternal-api/src/generated/java/datadog/trace/api/KnownTags.java Outdated

dougqh commented Jul 23, 2026
View reviewed changes

Comment threadinternal-api/src/generated/java/datadog/trace/api/KnownTags.java Outdated

dougqh commented Jul 23, 2026
View reviewed changes

Comment threadinternal-api/src/generated/java/datadog/trace/api/KnownTags.java Outdated

dougqh force-pushed the dougqh/generator-v2 branch 2 times, most recently from 7f3d0b1 to e6800b8Compare July 23, 2026 02:42

dougqh changed the title ~~Generate KnownTags from tag-conventions via the tag-registry code generator~~Generate KnownTags from tag-conventions via the tag-registry code generator (phase 2)Jul 23, 2026

dougqh force-pushed the dougqh/bloom-v2 branch from d6e7088 to 2bc5ae7Compare August 17, 2026 23:20

dougqh force-pushed the dougqh/generator-v2 branch from e6800b8 to 9b7c3d1Compare August 17, 2026 23:20

dougqh mentioned this pull request Aug 17, 2026
Encode tag ids with a colored slot and collapse dense presence to one occupancy long (phase 2) #12046
Draft

chatgpt-codex-connectorBot reviewed Aug 18, 2026
View reviewed changes

dougqh mentioned this pull request Aug 18, 2026
Map OpenTelemetry tag names via the registry (pass-through default) #12230
Draft

dougqhand others added 6 commits August 18, 2026 15:50

dougqh force-pushed the dougqh/bloom-v2 branch from 2bc5ae7 to bb9ebcaCompare August 18, 2026 20:14

dougqh force-pushed the dougqh/generator-v2 branch from 19222e1 to 33f32c4Compare August 18, 2026 20:14

		this.traceDenseTagsEnabled =
		configProvider.getBoolean(TracerConfig.TRACE_DENSE_TAGS_ENABLED, false);

		val outDir = destinationDirectory.get().asFile
		TagRegistryGenerator.generate(domainYaml.get().asFile, overlayYaml.get().asFile, outDir)

Uh oh!

Conversation

dougqh commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Generator (buildSrc)

internal-api

Verification

Stack

Non-goals (follow-on)

Uh oh!

datadog-prod-us1-4Bot commented Jul 23, 2026• edited by datadog-datadog-prod-us1-2 BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bits has a CI fix ready

Uh oh!

datadog-prod-us1-4Bot commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-stsBot commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dougqh commented Aug 18, 2026

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connectorBotAug 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotAug 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotAug 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotAug 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connectorBotAug 18, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh commented Aug 18, 2026

Uh oh!

chatgpt-codex-connectorBot commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dougqh commented Jul 23, 2026•
edited
Loading

datadog-prod-us1-4Bot commented Jul 23, 2026•
edited by datadog-datadog-prod-us1-2 Bot
Loading

datadog-prod-us1-4Bot commented Jul 23, 2026•
edited
Loading

dd-octo-stsBot commented Jul 23, 2026•
edited
Loading