fix(otel): preserve all exemplars per data point (#1662)

[prabhaks]

What

Fixes#1662.

insert_exemplars in src/otel/metrics.rs wrote every exemplar of a data point to the same static columns (exemplar_time_unix_nano, exemplar_span_id, exemplar_trace_id, exemplar_value). Since the keys were shared, each exemplar overwrote the previous one and only the last survived. This affected Sum, Histogram and ExponentialHistogram (and Gauge, which shares the same NumberDataPoint path).

Approach

Following the recommended option 1 in the issue, all exemplars for a data point are now serialized into a single nested exemplars array column. Each element is an object with exemplar_time_unix_nano, exemplar_span_id, exemplar_trace_id, exemplar_value and any filtered attributes. This matches the OTel data model (exemplars are a repeated field per data point) and keeps the schema stable regardless of exemplar count.

Backward compatibility

Added a P_OTEL_FLATTEN_EXEMPLARS flag (default false). When set to true the previous flat exemplar_* columns are emitted (last exemplar wins, same as before) for anyone already querying those columns. Per the discussion on the issue, the default is the new array behavior.

Bonus fix

Previously exemplar attributes were written straight into the data point map, so they leaked into compute_series_hash and fragmented physical-series identity. exemplars is now part of OTEL_METRICS_KNOWN_FIELD_LIST, so exemplar contents no longer affect the series hash.

Query validation (PoC)

Before settling on the schema I ran a PoC against the same library versions used here (datafusion 53, arrow/arrow-json 58) to confirm the query layer handles the array-of-objects column. arrow-json infers it as List<Struct<...>>, and DataFusion handles reading the column, array_length, unnest, struct field access, and filtering on nested fields. Details are in the issue thread.

Testing

• cargo test: 311 passed, 0 failed.
• New unit tests cover: all exemplars preserved in the array, legacy flat mode keeps last, empty exemplars insert nothing, end to end through a Sum, exemplar contents do not affect the series hash, and the known-field list update.

Flag reference

P_OTEL_FLATTEN_EXEMPLARS env var / --otel-flatten-exemplars flag, default false (new nested exemplars array). Set to true for the legacy flat exemplar_* columns.

Summary by CodeRabbit

• New Features
  • Added --otel-flatten-exemplars / P_OTEL_FLATTEN_EXEMPLARS to control how OTEL metric exemplars are represented when flattening.
  • Default preserves all exemplars in a nested exemplars structure; enabling uses legacy flat exemplar_* fields (“last wins”).
• Bug Fixes
  • Metric series identity/hashing no longer changes based on exemplar contents.
• Tests
  • Added regression coverage for nested vs legacy behavior, empty exemplars, end-to-end flattening, and series-hash invariance.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

OTEL metrics now support configurable exemplar storage: nested arrays preserve all exemplars by default, while P_OTEL_FLATTEN_EXEMPLARS enables legacy flattened columns. The mode flows through JSON and protobuf processing, with updated series identity handling and regression tests.

Changes

OTEL exemplar handling

Layer / File(s)	Summary
Configuration and series identity contract src/cli.rs, src/otel/metrics.rs	Adds the otel_flatten_exemplars option and registers exemplars as a known OTEL metric field.
Exemplar serialization and flattening src/otel/metrics.rs	Adds JSON conversion helpers and applies nested-array or legacy flattened exemplar output across gauge, sum, histogram, and exponential-histogram records.
Pipeline wiring and validation src/otel/metrics.rs	Propagates the option through JSON and protobuf processing and tests preservation, legacy overwrite behavior, empty exemplars, and series hashes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
participant OTELClient
participant flatten_otel_metrics
participant process_resource_metrics
participant flatten_metrics_record
participant insert_exemplars
OTELClient->>flatten_otel_metrics: OTEL metrics payload
flatten_otel_metrics->>process_resource_metrics: configured exemplar mode
process_resource_metrics->>flatten_metrics_record: metric record and mode
flatten_metrics_record->>insert_exemplars: exemplar list and mode
insert_exemplars-->>flatten_metrics_record: nested array or flattened columns
flatten_metrics_record-->>process_resource_metrics: flattened records

Loading

Suggested reviewers:nikhilsinhaparseable

Poem

I’m a rabbit with metrics to spare,
Nesting exemplars with careful care.
Or flatten them wide, old-style in a row,
While every little trace gets a place to show.
Hop, hop—the data stays whole!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the core fix: preserving all OTEL exemplars per data point.
Description check	✅ Passed	The description covers the issue, approach, compatibility flag, testing, and flag reference, with only minor template sections omitted.
Linked Issues check	✅ Passed	The changes match #1662 by preserving exemplars in a nested array, supporting legacy flat mode, and covering affected metric types.
Out of Scope Changes check	✅ Passed	No obvious unrelated code changes are present beyond the exemplar preservation fix and its related compatibility and hashing updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[prabhaks]

Note on the failing coverage check: it fails at the clippy step because of pre-existing lints on main from the newer Rust/clippy 1.97, not because of this PR (none of the flagged files are touched here). I opened #1721 to fix those lints and unblock CI. Once #1721 is merged I will rebase this branch on top so its checks go green.

[nikhilsinhaparseable]

@prabhaks thank you for raising the lint PR, i have verified and merged it, you can rebase this PR now

[prabhaks]

@prabhaks thank you for raising the lint PR, i have verified and merged it, you can rebase this PR now

Done, thank you for reviewing and merging my other PR!

[prabhaks]

@nikhilsinhaparseable@parmesant please take a look at this PR whenever you get a chance!

[prabhaks]

@parmesant@nikhilsinhaparseable Any thoughts on this fix?

[parmesant]

Hey @prabhaks
Got caught up with another issue today. I'll pick this one up tomorrow

[parmesant]

@prabhaks the PR looks good but there is a doubt that we have-

• While ingesting data from astronomy shop, we could see that a new column called exemplars_exemplar_value_list got created which contained the array of values and another column exemplars_exemplar_value which is empty. Is that intended?

[prabhaks]

@prabhaks the PR looks good but there is a doubt that we have-

• While ingesting data from astronomy shop, we could see that a new column called exemplars_exemplar_value_list got created which contained the array of values and another column exemplars_exemplar_value which is empty. Is that intended?

Hi @parmesant - I am on vacation till July 22nd. Will get back to you after that!

[parmesant]

Have a great vacation!

[prabhaks]

Good catch, and no, that empty column is not intended.

Root cause: exemplar values come through as an integer for AsInt exemplars and a float for AsDouble. Once the exemplars array is flattened on ingest, each field becomes a list column, so exemplars_exemplar_value infers as List<Int64> for some batches and List<Float64> for others. When the inferred type differs from what the stream already stored, the conflict handling routes that batch into a typed sibling column, which is where exemplars_exemplar_value_list comes from, and the original exemplars_exemplar_value is left empty. The astronomy shop mixes int and double exemplars, so it runs into this.

I reproduced it with a small test: an int-typed exemplar batch followed by a double-typed one yields exactly {"exemplars_exemplar_value": "exemplars_exemplar_value_list"} as a conflict, while double followed by double yields no conflict.

Fix (just pushed): normalize both int and double exemplar values to f64 in the flattener, so exemplars_exemplar_value is always List<Float64>. That gives a single clean column with no empty sibling. The other exemplar fields (span_id, trace_id, time) are strings and were already stable.

One heads up for re-testing: the earlier run already created the empty exemplars_exemplar_value and the exemplars_exemplar_value_list columns in that stream's schema, so you will want to test against a fresh stream (or recreate it) to avoid looking at the columns from the old schema.

For context on the overall shape: the exemplars array flattens into parallel per-field list columns (exemplars_exemplar_time_unix_nano, exemplars_exemplar_span_id, exemplars_exemplar_trace_id, exemplars_exemplar_value, plus any filtered attributes), correlated by position within each data point. Happy to adjust the layout if you would prefer something different.

[prabhaks]

@parmesant Can you please test against a fresh stream!

[parmesant]

Sure
Will do

[prabhaks]

Sure Will do

thank you @parmesant , please let me know your findings!

[parmesant]

@prabhaks the PR looks good. Just made one small change to exclude the flattened exemplars_* columns from series hash calculations

[prabhaks]

@prabhaks the PR looks good. Just made one small change to exclude the flattened exemplars_* columns from series hash calculations

thanks @parmesant for verifying and making the change! If everything looks fine, please approve the PR, so that I can merge it!