Skip to content

Add hidden metrics and MAX/MIN/incremental derived metric aggregation - #13505

Merged
cmcfarlen merged 12 commits into
apache:masterfrom
cmcfarlen:metrics-hidden-and-derived
Aug 18, 2026
Merged

cmcfarlen merged 12 commits into
apache:masterfrom
cmcfarlen:metrics-hidden-and-derived

Conversation

@cmcfarlen

@cmcfarlen cmcfarlen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Adds two facilities to ts::Metrics, plus a traffic_ctl option to inspect the first, and fixes three pre-existing defects found along the way. A follow-up PR uses these for per-upstream-server connection metrics; this PR is independently useful and stands on its own.

Hidden metrics

A second Storage instance, reached through Metrics::hidden_instance(), for high-cardinality intermediate values that are worth recording but not worth publishing. Gauge::createHiddenPtr / Counter::createHiddenPtr return the same correctly typed pointer as createPtr, so a hidden metric is read and written with the ordinary typed mutators with no cast at the call site.

A separate store rather than a per-metric "hidden" flag is deliberate: it makes hidden metrics structurally unreachable from the published store, so no consumer can expose one by forgetting to check a flag.

Since that also makes them hard to debug, traffic_ctl metric match --include-hidden lists them. The rec type bit for this (RECT_HIDDEN_METRIC = 0x40) sits deliberately outside RECT_ALL (0x3F), so hidden metrics are returned only when explicitly asked for and never as a side effect of a broad query.

Derived metric aggregation

  • MAX and MIN in addition to SUM. The accumulator is seeded from the first source rather than from zero, since a zero seed is only correct for SUM and would clamp MIN to <= 0.
  • Derived::add_source(), for aggregates whose sources are discovered while the process runs rather than known at startup. Repeatedly calling derive() for one derived name does not work for this: it appends a separate entry per call, all targeting the same metric, so each update overwrites the others with its own subset and the last writer silently wins. add_source() accumulates into a single entry, and re-registering an existing source is a no-op.

Pre-existing defects fixed

  • Storage::create() had no exhaustion check. Filling the last blob let the following bookkeeping call addBlob() and write one past the end of _blobs. Verified by temporarily shrinking MAX_BLOBS: the current code segfaults, and the debug_assert in addBlob() does not catch it because it is off by one against the access it guards (_blobs[++_cur_blob]) — and being a debug_assert, it is compiled out of release builds entirely. Now a release_assert against MAX_BLOBS - 1, with create() refusing the final slot and returning the reserved bad_id.
  • Unresolvable derived sources were not skipped. A source given by name or id that does not resolve was still passed to lookup(), which masks the unresolved id down to the reserved bad_id slot, so the aggregate silently included that slot's value. Observable under MAX/MIN, where the bad_id value can become the winning one; under SUM it was hidden by bad_id holding zero.
  • The metrics unit tests were order dependent. They asserted absolute metric ids and iterator positions, which only hold when the case runs first against an otherwise empty store. Any other case that creates a published metric made them fail. This lands first so the fragility is never introduced.

Testing

  • Unit tests grow from 487 assertions / 32 cases to 6208 / 36, and pass under --order rand across many seeds. Each fix was checked against the unfixed code first to confirm the new assertions actually discriminate.
  • tests/gold_tests/jsonrpc/metric_match_include_hidden.test.py covers the --include-hidden RPC round trip end to end. This is load-bearing: the rec type also has to be accepted by the JSONRPC request decoder, which validates each requested type against a whitelist and rejects the entire request otherwise. That was invisible to every build-level check and only showed up end to end.
  • Verified against a running traffic_server with two temporary hidden metrics registered: absent from metric match, present with --include-hidden, and absent from a broad proxy.process query of ~900 published metrics.
  • Docs build clean under -W with nitpicky = True.

Co-authored-by: @serrislew

The tests asserted absolute metric ids and iterator positions, which only
hold when the case runs first against an otherwise empty store. Any other
test case that creates a published metric makes them fail. Assert on
relative state instead so the cases can run in any order.
Hidden metrics are stored but never published. Using a separate Storage
instance rather than a per-metric flag makes them structurally unreachable
from the published store, so no metric consumer can expose them by
omission.

Gauge and Counter each gain createHiddenPtr overloads which return the
same correctly typed pointer as createPtr, so a hidden metric is read and
written with the normal typed mutators and no cast is needed at the call
site.
Storage::create() had no exhaustion check, so filling the last blob let the
following bookkeeping call addBlob() and write one past the end of _blobs.
Refuse the final slot instead and return the reserved bad_id, which keeps
addBlob() from ever being reached in a full store and costs one slot out of
8M.

The guard in addBlob() was also off by one against the access it protects,
since the write is to _blobs[++_cur_blob], and being a debug_assert it was
compiled out of release builds entirely. Make it a release_assert against
MAX_BLOBS - 1.
Hidden metrics are invisible to normal queries by design, which makes them
hard to debug. Add an opt-in rec type bit, deliberately outside RECT_ALL so
hidden metrics are never returned unless explicitly requested.

The rec type also has to be accepted by the JSONRPC request decoder, which
validates each requested type against a whitelist and rejects the whole
request otherwise. No wire or schema change is needed, as rec_types is
already an untyped list of ints.
Derived metrics could only sum their sources. Add an op to the spec so a
derived metric can also take the max or min across its sources, which is
what an aggregate over instantaneous gauges needs.

The accumulator is seeded from the first source rather than from zero,
since a zero seed is only correct for SUM and would clamp MIN to <= 0. op
defaults to SUM, so existing specs are unaffected.
A source given by name or id that does not resolve was still passed to
lookup(), which masks the unresolved id down to the reserved bad_id slot.
The aggregate then silently included that slot's value instead of skipping
the source, with no error reported.

Resolve each source first and skip it if it does not resolve. This is
observable under MAX and MIN, where the bad_id value can become the winning
one; under SUM it happened to be hidden by bad_id holding zero.
derive() only accepts a fixed initializer_list, which does not work for
aggregates whose sources are discovered as the process runs. Calling it
repeatedly for one derived name does not help either: it appends a separate
entry per call, all targeting the same metric, so each update overwrites the
others with its own subset and the last writer silently wins.

add_source() accumulates sources into a single entry instead. Registering a
source that is already present is a no-op, so a caller that may re-register
the same source need not track that itself.
Add a developer guide page for the metrics registry covering the hidden
store, how it differs from the published one and why it is a separate store
rather than a flag, and the derived metric aggregation ops including when
derived values are recomputed and what that means for a sampled maximum.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends ts::Metrics with (1) a separate hidden-metrics store for high-cardinality internal values and (2) richer derived-metric aggregation (SUM/MAX/MIN plus incremental source registration), and wires this through traffic_ctl metric match --include-hidden via JSONRPC, with accompanying tests and documentation.

Changes:

  • Add Metrics::hidden_instance() plus Gauge::createHiddenPtr / Counter::createHiddenPtr to record internal metrics that are structurally unreachable from the published registry.
  • Enhance derived metrics with Op { SUM, MAX, MIN }, correct accumulator seeding, add Derived::add_source(), and fix handling of unresolved sources.
  • Add traffic_ctl metric match --include-hidden end-to-end support (including JSONRPC request decoding), plus unit + gold tests and new internal library docs.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/gold_tests/jsonrpc/metric_match_include_hidden.test.py New gold test validating JSONRPC accepts the new rec type and traffic_ctl flag.
src/tsutil/unit_tests/test_Metrics.cc Expands Metrics unit coverage; removes order-dependent assertions; adds tests for ops/add_source/hidden store.
src/tsutil/Metrics.cc Implements hidden instance; fixes blob exhaustion; adds derived ops + add_source and unresolved-source skipping.
src/traffic_ctl/traffic_ctl.cc Adds --include-hidden option and usage for traffic_ctl metric match.
src/traffic_ctl/CtrlCommands.h Extends record_fetch signature to accept an include-hidden flag; adds option key constant.
src/traffic_ctl/CtrlCommands.cc Passes --include-hidden into the JSONRPC record lookup request.
src/records/RecCore.cc Adds hidden-metric enumeration to regex record lookup.
src/mgmt/rpc/handlers/records/Records.cc Allows RECT_HIDDEN_METRIC in JSONRPC request decoding (opt-in).
include/tsutil/Metrics.h Public API additions for hidden metrics and derived ops/add_source.
include/shared/rpc/RPCRequests.h Adds METRIC_REC_TYPES_INCLUDE_HIDDEN for include-hidden requests.
include/records/RecDefs.h Defines RECT_HIDDEN_METRIC = 0x40 deliberately outside RECT_ALL.
doc/developer-guide/internal-libraries/Metrics.en.rst New internal library documentation for hidden + derived metrics behavior and constraints.
doc/developer-guide/internal-libraries/index.en.rst Adds Metrics to internal libraries index.
doc/appendices/command-line/traffic_ctl.en.rst Documents traffic_ctl metric match --include-hidden.

Comment thread src/records/RecCore.cc Outdated
@cmcfarlen cmcfarlen self-assigned this Aug 6, 2026
@cmcfarlen cmcfarlen added this to the 11.0.0 milestone Aug 6, 2026
@bryancall
bryancall self-requested a review August 10, 2026 22:36
The record lookup callback rejects any record whose rec_type shares no bit
with the requested mask, so tagging hidden metrics RECT_PROCESS alone made
a request for only RECT_HIDDEN_METRIC fail with REQUESTED_TYPE_MISMATCH.
Now that the request decoder accepts that type on its own, such a request
is expressible, so set both bits. Add a unit test covering the hidden-only
and include-hidden requests and confirming RECT_ALL still excludes them.
Copilot AI review requested due to automatic review settings August 13, 2026 00:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Comment thread src/records/RecCore.cc Outdated
bryancall
bryancall previously approved these changes Aug 18, 2026

@bryancall bryancall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through all 16 files. This is a clean change and the commit hygiene is genuinely good: one concern per commit, each message stating the defect and the reasoning, with the test order-independence fix landed first so the fragility never exists in the history.

A few things worth calling out as done right, because they were easy to get wrong:

  • A separate Storage rather than a per-metric "hidden" flag. That makes hidden metrics structurally unreachable from the published store instead of relying on every consumer remembering to check a bit.
  • RECT_HIDDEN_METRIC = 0x40 sitting outside RECT_ALL = 0x3F, so no existing broad query changes behavior, and the JSONRPC decoder change stays on the read-only lookup path.
  • Seeding the derived value from the first source instead of from zero. A zero seed is correct only for SUM and would clamp every MIN result to <= 0. The guard tests are built so an aliased-in bad_id zero would change the result, strictly positive sources for MIN and strictly negative for MAX, so they actually discriminate.
  • The developer guide page documents the sampling semantics of a derived MAX, the 5 second recompute interval, and the hazard that ids are not portable across stores.

One general note that did not fit on a line. test_RecHiddenMetricLookup.cc re-implements the requested-type check locally rather than calling the real one, so nothing verifies end to end that a hidden metric survives the check in get_record_regex_impl. If that check changed from (recType & rec_type) == 0 to an equality comparison, this test would still pass, and the gold test would not catch it either because a live traffic_server registers no hidden metrics for the query to return. The mirrored check is documented with a comment pointing at the original and the substance is asserted, so the risk is small. I raise it only because the PR body correctly identifies this interaction as the one thing invisible to every build-level check.

Everything I found is non-blocking. Four inline comments below.

Comment thread src/records/RecCore.cc Outdated
Comment thread src/records/RecCore.cc Outdated
Comment thread src/tsutil/Metrics.cc
Comment thread doc/appendices/command-line/traffic_ctl.en.rst
Both lookup functions build a RecRecord on the stack for metrics, which
live outside the g_records array, and hand it to the caller's callback. The
JSONRPC encoder reads version, registered, rsb_id, order and data_default
unconditionally, so leaving them indeterminate lets a --format json metric
query emit different values on successive runs, and reading an
indeterminate bool is undefined behavior.

Five sites, all with the same one word fix. Only the hidden metric loop is
new in this branch; the rest have had the pattern for years.
createSpan checked whether a span fit before reserving it but never
re-checked afterwards, so a span ending exactly on MAX_SIZE left the offset
at MAX_SIZE with no new blob allocated. The next create() then wrote one
past the end of that blob's name array, and end() became an id that
iterator::next() can never reach, since it wraps at ++offset == MAX_SIZE.
create() has always grown as soon as it consumed the last slot; do the
same here.

Also refuse a span that would fill or overflow the final blob, so the new
growth cannot ask addBlob() to go past the last one and trip its assert.

createSpan(MAX_SIZE) always starts a fresh blob and fills it exactly,
whatever the current offset, so the added test reaches the boundary
deterministically. It fails without the fix.
Three small corrections to the flag added earlier in this branch:

Skip slot 0 when walking the hidden store. Every Storage reserves it for
the bad_id placeholder, so it exists under the same name in both stores and
a query matching it returned two records differing only in value, in
exactly the debugging situation the flag is for.

Scope the option to 'match' with a nested program directive. As a bare
option under 'traffic_ctl metric' it rendered as a peer of get, match and
describe, so it read as another subcommand rather than a flag on match.
This follows the 'config get --records' pattern earlier in the file.

Put the flag before the positional in the CLI example usage so it agrees
with that synopsis, which is also the convention the rest of the file uses.
Copilot AI review requested due to automatic review settings August 18, 2026 17:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@cmcfarlen
cmcfarlen requested a review from bryancall August 18, 2026 19:46

@bryancall bryancall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the three new commits. All four of my comments were addressed, two of them beyond what I asked for. Approving again.

C1, value-initializing the synthetic records: fixed, and over-delivered. I asked for the hidden-metric site and suggested the published one. You found all five, including both RecLookupRecord sites and the RECD_STRING StaticString path, which is the one that actually reads data_default.rec_string. I checked that {} genuinely zeroes everything here: RecRecord is an aggregate with no user-provided constructors, RecStatMeta and RecConfigMeta are both plain types so the anonymous union zero-initializes wholesale, and RecMutex is itself an aggregate that gets zeroed before any member constructor runs. The synthetic records' mutex is never acquired on these paths, so the all-zero pthread_mutex_t is never used.

C2, the bad_id duplicate: fixed correctly. The unconditional ++it is safe for a reason worth having written down, and your comment writes it down: Storage's constructor unconditionally allocates blob 0 and asserts that create("proxy.process.api.metrics.bad_id") returns 0, so begin() != end() always holds and a hidden store containing only bad_id lands the single increment exactly on end().

C3, the createSpan boundary: the case I named is closed, but the new guard does not do what its comment says. Details inline. Short version: createSpan can now advance the blob index twice in one call, and the guard only inspects _cur_blob as it is on entry.

C4, the docs: half fixed. The flag ordering now agrees between the rst synopsis and the help string, and the .. program:: addition correctly rescopes the cross-reference target without breaking the following describe and get options. The rendering problem I was actually complaining about is still there, though. Inline.

I am approving rather than holding this, because the one real defect needs roughly 8.38 million live published metrics to reach and createSpan has no callers in the tree outside the unit tests. I want to be clear that I am approving on reachability, not because C3 and C4 are fully resolved.

One claim from my own earlier reasoning that I should retract: I suggested the fix might be to make the trailing addBlob() conditional. That would not reintroduce the out-of-bounds names[1024] write, since create()'s own guard already catches that state, but it would reintroduce the unreachable-end() half. So the warning against it stands, just for a different reason than I had in mind.

Comment thread src/tsutil/Metrics.cc
// On the final blob there is nowhere left to grow, so refuse a span that would fill or overflow
// it rather than letting addBlob() assert. Same intent as the guard in create(), and the same
// cost: some slots of the last blob go unused.
if (_cur_blob >= MAX_BLOBS - 1 && _cur_off + size >= MAX_SIZE) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard does not cover the case its comment describes, and the trailing addBlob() you added below makes a state that was previously survivable into a release abort.

createSpan can now advance the blob index twice in one call: once at line 201 because a span has to be contiguous, and once at line 220 when it ends exactly on the boundary. The guard only inspects _cur_blob as it is on entry, so it misses the case where the first advance is what puts you on the last blob.

Walking _cur_blob == MAX_BLOBS - 2 (8190), _cur_off == 500, size == MAX_SIZE:

  • line 193: 8190 >= 8191 is false, so no refusal
  • line 201: 500 + 1024 > 1024, so addBlob() runs, its release_assert(8190 < 8191) passes, leaving _cur_blob == 8191 and _cur_off == 0
  • line 214: _cur_off becomes exactly 1024
  • line 220: 1024 >= 1024, so addBlob() runs again and release_assert(8191 < 8191) fires

release_assert is defined outside the debug gate, so that aborts traffic_server in a release build. Sweeping _cur_blob in [8188, 8191] against every _cur_off and size gives exactly 1023 aborting triples, all at _cur_blob == 8190, _cur_off in [1, 1023], size == MAX_SIZE. The same sweep against the pre-commit body aborts zero times at 8190, so this state is newly fatal rather than a pre-existing hole.

Reachability is why this is not a blocker: it needs about 8.38M live published metrics, roughly half a gigabyte of blob storage plus an 8.4M-entry lookup map, and createSpan has no non-test callers. It is still worth closing, because the guard reads as complete and is not.

The obvious rewrite in terms of a target blob and offset is wrong, so I want to save you the detour. With _cur_blob == 8191, _cur_off == 500, size == 600 it computes target_off == 0, fails the >= MAX_SIZE test, and falls through to an addBlob() that the current guard correctly refuses. Counting the advances instead subsumes the existing behavior:

// A span can advance the blob index twice: once because it must be contiguous, and once more
// when it ends exactly on the boundary. Refuse unless both advances are available.
const bool     needs_fresh_blob = (_cur_off + size > MAX_SIZE);
const size_t   end_off          = (needs_fresh_blob ? 0 : _cur_off) + size;
const unsigned needed           = (needs_fresh_blob ? 1u : 0u) + (end_off >= MAX_SIZE ? 1u : 0u);

if (_cur_blob + needed > MAX_BLOBS - 1) {
  if (id) {
    *id = 0; // Slot 0 is the reserved bad_id.
  }
  return {};
}

That turns the 1023 aborting triples into graceful bad_id refusals, never leaves _cur_off == MAX_SIZE, and matches the current guard on every case it already handles.

the given regular expression.

.. program:: traffic_ctl metric match
.. option:: --include-hidden

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the cross-reference scope, which is real and worth having, but not the thing I was complaining about. --include-hidden is still a column-0 .. option:: directive, so it still renders as a top-level sibling of get, match, describe and monitor rather than as a flag belonging to match.

The file already establishes the pattern 45 lines up: ssl-multicert show nests its .. option:: --yaml, -y inside the body of the subcommand option. Indenting this block three spaces so it sits inside match gets the rendering right, and makes both .. program:: lines here unnecessary.

metric_command.add_command("match", "Get metrics matching a regular expression", "", MORE_THAN_ZERO_ARG_N, Command_Execute);
metric_command.add_command("match", "Get metrics matching a regular expression", "", MORE_THAN_ZERO_ARG_N, Command_Execute)
.add_option("--include-hidden", "", "Also match hidden (internal, normally unpublished) metrics")
.add_example_usage("traffic_ctl metric match [--include-hidden] METRIC [METRIC ...]");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag position now agrees with the rst, which is what I asked for. The operand name does not: this says METRIC while traffic_ctl.en.rst:952 says REGEX, and the subcommand's own description one line up is "Get metrics matching a regular expression".

METRIC is the misleading one here, since metric get genuinely takes literal names and this reads as if match does too. The sibling command at line 124 spells it traffic_ctl config match [OPTIONS] REGEX [REGEX ...]. Suggest traffic_ctl metric match [--include-hidden] REGEX [REGEX ...].

Comment thread src/records/RecCore.cc
// Slot 0 of every Storage is the reserved bad_id placeholder, so it exists under the same name
// in both stores. Skip it here, otherwise a query matching it returns two identically named
// records that differ only in value.
auto it = hidden.begin();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skip is correct and solves the duplicate. Three follow-on notes, none blocking.

Skipping slot 0 removes the last way to see the hidden store's bad_id counter from the CLI. RecLookupRecord consults only ts::Metrics::instance(), never hidden_instance(), so metric get cannot reach it either. That counter accumulates every failed createHiddenPtr, and the new Metrics.en.rst tells operators that unbounded hidden metric names are the likely cause of exhaustion, so the diagnostic the docs point at is now unreadable. Emitting it under a disambiguated name, appending a .hidden suffix for the hidden store's slot 0, would resolve the duplicate confusion and keep the signal.

Moving from a range-for to an explicit loop also changed end() from evaluated once to evaluated per iteration, so this now picks up entries registered by another thread mid-walk where the published loop 30 lines up does not. I checked termination and it is fine, but the two loops now have different snapshot semantics for no stated reason. auto const stop = hidden.end(); would keep them consistent.

There is also no test for the skip, and it is implemented as manual iterator arithmetic rather than a condition inside the loop, which is the shape a later refactor back to a range-for quietly undoes. A section in test_RecHiddenMetricLookup.cc calling RecLookupMatchingRecords with RECT_PROCESS | RECT_HIDDEN_METRIC and a bad_id pattern, asserting a count of 1, is about four lines and fails pre-fix with a count of 2.


TEST_CASE("Metrics span lands exactly on a blob boundary", "[libtsapi][Metrics]")
{
// A span has to be contiguous, so createSpan(MAX_SIZE) always starts a fresh blob and then fills

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good test to add, and picking MAX_SIZE because it is the one size that reaches the boundary deterministically is the right instinct. Two notes.

The stated mechanism is not quite right. createSpan(MAX_SIZE) does not always start a fresh blob: when _cur_off == 0 on entry, _cur_off + size > MAX_SIZE is 1024 > 1024, which is false, so no addBlob() and the span is placed in the existing empty blob. The conclusion holds either way, so this is only comment accuracy, but the comment is the sole explanation of why MAX_SIZE is special and a reader who trusts it could conclude a smaller size behaves the same. The property that actually holds is that a span of MAX_SIZE always lands at offset 0 of an empty blob, growing one first if the current blob is partly used, and then fills it completely.

The test also does not fail deterministically against the pre-fix code. Without the trailing addBlob(), _cur_off is left at MAX_SIZE and the next create() writes names[1024], one past the end of a std::array<NameAndId, 1024>. That is undefined behavior a sanitizer catches, but in a plain build it usually lands in adjacent storage and the REQUIREs still pass, so this stops guarding the fix in any non-sanitizer lane. Asserting that the iterator can actually reach end() after the span would pin the other half directly, since an offset parked at MAX_SIZE makes end() unreachable for iterator::next().

@cmcfarlen
cmcfarlen merged commit 266ee96 into apache:master Aug 18, 2026
15 checks passed
@github-project-automation github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Aug 18, 2026
@cmcfarlen
cmcfarlen deleted the metrics-hidden-and-derived branch August 18, 2026 21:33
@cmcfarlen cmcfarlen moved this from For v10.2.0 to Picked v10.2.1 in ATS v10.2.x Aug 19, 2026
@cmcfarlen cmcfarlen modified the milestones: 11.0.0, 10.2.1 Aug 19, 2026
@cmcfarlen

Copy link
Copy Markdown
Contributor Author

Cherry-picked to 10.2.x

cmcfarlen added a commit that referenced this pull request Aug 19, 2026
…#13505)

* tsutil: make the Metrics unit tests order independent

The tests asserted absolute metric ids and iterator positions, which only
hold when the case runs first against an otherwise empty store. Any other
test case that creates a published metric makes them fail. Assert on
relative state instead so the cases can run in any order.

* tsutil: add a separate storage for hidden metrics

Hidden metrics are stored but never published. Using a separate Storage
instance rather than a per-metric flag makes them structurally unreachable
from the published store, so no metric consumer can expose them by
omission.

Gauge and Counter each gain createHiddenPtr overloads which return the
same correctly typed pointer as createPtr, so a hidden metric is read and
written with the normal typed mutators and no cast is needed at the call
site.

* tsutil: fail gracefully when metric storage is exhausted

Storage::create() had no exhaustion check, so filling the last blob let the
following bookkeeping call addBlob() and write one past the end of _blobs.
Refuse the final slot instead and return the reserved bad_id, which keeps
addBlob() from ever being reached in a full store and costs one slot out of
8M.

The guard in addBlob() was also off by one against the access it protects,
since the write is to _blobs[++_cur_blob], and being a debug_assert it was
compiled out of release builds entirely. Make it a release_assert against
MAX_BLOBS - 1.

* traffic_ctl: add --include-hidden to metric match

Hidden metrics are invisible to normal queries by design, which makes them
hard to debug. Add an opt-in rec type bit, deliberately outside RECT_ALL so
hidden metrics are never returned unless explicitly requested.

The rec type also has to be accepted by the JSONRPC request decoder, which
validates each requested type against a whitelist and rejects the whole
request otherwise. No wire or schema change is needed, as rec_types is
already an untyped list of ints.

* tsutil: support MAX and MIN aggregation for derived metrics

Derived metrics could only sum their sources. Add an op to the spec so a
derived metric can also take the max or min across its sources, which is
what an aggregate over instantaneous gauges needs.

The accumulator is seeded from the first source rather than from zero,
since a zero seed is only correct for SUM and would clamp MIN to <= 0. op
defaults to SUM, so existing specs are unaffected.

* tsutil: skip derived metric sources that do not resolve

A source given by name or id that does not resolve was still passed to
lookup(), which masks the unresolved id down to the reserved bad_id slot.
The aggregate then silently included that slot's value instead of skipping
the source, with no error reported.

Resolve each source first and skip it if it does not resolve. This is
observable under MAX and MIN, where the bad_id value can become the winning
one; under SUM it happened to be hidden by bad_id holding zero.

* tsutil: allow adding derived metric sources at runtime

derive() only accepts a fixed initializer_list, which does not work for
aggregates whose sources are discovered as the process runs. Calling it
repeatedly for one derived name does not help either: it appends a separate
entry per call, all targeting the same metric, so each update overwrites the
others with its own subset and the last writer silently wins.

add_source() accumulates sources into a single entry instead. Registering a
source that is already present is a no-op, so a caller that may re-register
the same source need not track that itself.

* doc: document hidden and derived metrics

Add a developer guide page for the metrics registry covering the hidden
store, how it differs from the published one and why it is a separate store
rather than a flag, and the derived metric aggregation ops including when
derived values are recomputed and what that means for a sampled maximum.

* Tag hidden metrics with RECT_HIDDEN_METRIC as well as RECT_PROCESS

The record lookup callback rejects any record whose rec_type shares no bit
with the requested mask, so tagging hidden metrics RECT_PROCESS alone made
a request for only RECT_HIDDEN_METRIC fail with REQUESTED_TYPE_MISMATCH.
Now that the request decoder accepts that type on its own, such a request
is expressible, so set both bits. Add a unit test covering the hidden-only
and include-hidden requests and confirming RECT_ALL still excludes them.

* Value-initialize the synthetic records in the record lookups

Both lookup functions build a RecRecord on the stack for metrics, which
live outside the g_records array, and hand it to the caller's callback. The
JSONRPC encoder reads version, registered, rsb_id, order and data_default
unconditionally, so leaving them indeterminate lets a --format json metric
query emit different values on successive runs, and reading an
indeterminate bool is undefined behavior.

Five sites, all with the same one word fix. Only the hidden metric loop is
new in this branch; the rest have had the pattern for years.

* Grow a new blob when a span ends on the blob boundary

createSpan checked whether a span fit before reserving it but never
re-checked afterwards, so a span ending exactly on MAX_SIZE left the offset
at MAX_SIZE with no new blob allocated. The next create() then wrote one
past the end of that blob's name array, and end() became an id that
iterator::next() can never reach, since it wraps at ++offset == MAX_SIZE.
create() has always grown as soon as it consumed the last slot; do the
same here.

Also refuse a span that would fill or overflow the final blob, so the new
growth cannot ask addBlob() to go past the last one and trip its assert.

createSpan(MAX_SIZE) always starts a fresh blob and fills it exactly,
whatever the current offset, so the added test reaches the boundary
deterministically. It fails without the fix.

* Polish the --include-hidden surface

Three small corrections to the flag added earlier in this branch:

Skip slot 0 when walking the hidden store. Every Storage reserves it for
the bad_id placeholder, so it exists under the same name in both stores and
a query matching it returned two records differing only in value, in
exactly the debugging situation the flag is for.

Scope the option to 'match' with a nested program directive. As a bare
option under 'traffic_ctl metric' it rendered as a peer of get, match and
describe, so it read as another subcommand rather than a flag on match.
This follows the 'config get --records' pattern earlier in the file.

Put the flag before the positional in the CLI example usage so it agrees
with that synopsis, which is also the convention the rest of the file uses.

(cherry picked from commit 266ee96)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Picked v10.2.1

Development

Successfully merging this pull request may close these issues.

3 participants