Retract per server metrics when the mode changes

[cmcfarlen]

Uses ts::Metrics::unlist from #13616, now merged, so this is rebased onto it and the diff is its own.

Problem

proxy.config.http.per_server.connection.metric_aggregate is RECU_DYNAMIC and overridable, but the decision it drives — which per server metric names get published — was made once, in the ConnectionTracker::Group constructor, and a published metric name could not be withdrawn. So the setting only ever took effect for names created after it changed.

Seen in production. A box that ran for a while at metric_aggregate 0 before being switched to 2 reports both shapes, and no reload removes the first set:

proxy.process.http.per_server.current_connection.ocsp.apple.com.17.253.67.133:80 0
...
proxy.process.http.per_server.current_connection.ocsp.apple.com 0
proxy.process.http.per_server.current_connection.max.ocsp.apple.com 0

The metric store hands out ids in allocation order and traffic_ctl prints them that way, so the dump is a timeline: every <fqdn>.<ip>:<port> name was created before the first aggregate name, with no interleaving. The config change took effect for everything after it; what came before was unretractable.

What the modes mean now

The suppressed-per-group mode was specified as a single metric per hostname — the max — not the sums as well. Mode 2 is that, and mode 3 is new for when the totals are wanted too:

value	per group	sums	max
0 AGGREGATE_NONE	published	no	no
1 AGGREGATE_GROUP	published	yes	yes
2 AGGREGATE_MAX	hidden	no	yes
3 AGGREGATE_SUM	hidden	yes	yes

Mode 1 is unchanged. AGGREGATE_ONLY is gone; 2 and 3 replace it.

The constructor now reduces to three independent decisions — publish the sums, publish the max, publish the per group metrics — each of which either registers a derived source or unlists the name. That reads better than the nested condition it replaces, and it is what makes 3 → 2 withdraw the sums rather than leave them behind.

An out of range value from a plugin is normalised to AGGREGATE_GROUP once, at the top of the constructor, rather than being implicit in the conditions.

Metric rename

current_connection_max becomes current_connection.max. ATS separates a qualifier with a dot — proxy.process.eventloop.time.max, .events.max — not an underscore. The metric only exists on master, from #13506, so the rename is free now and would not be after a release carries it.

What converges, and when

A change is applied per group, when that group is next constructed, which happens on the first connection after its count last fell to zero. Group::release() is called from PoolableSession::release_outbound_connection_tracking(), so it is the upstream session closing that erases the group, not the transaction ending. With origin keep alive on, a pooled session holds a group open and that group keeps whatever setting it was built with; a group that never goes idle never re-evaluates.

Two consequences worth knowing rather than discovering:

• The sums are named per hostname, not per group, so where the mappings for one hostname disagree about this setting, the last group constructed decides whether they are published.
• Retraction is not immediate. It follows session churn.

Both are documented at the enum and in records.yaml.en.rst.

Tests

src/iocore/net/unit_tests/test_ConnectionTracker.cc is new. It drives the production sequence in process: run at AGGREGATE_NONE so the per group names publish, switch, open and close another connection, assert the names are gone and the aggregates are there. Nine sections cover each mode, both switch directions for the sums, and the no-aggregate fallback at 2 and 3.

Getting that harness right took a correction worth recording: TxnState::release() only decrements, so a test using it never erases the group and nothing is re-evaluated. It has to follow the real path — TxnState::drop() into the session, then Group::release().

per_server_connection_max.test.py gains AggregateRetractionTest, which drives traffic at 0, asserts the per group name is published so the later assertion cannot pass vacuously, raises the setting with traffic_ctl, drives traffic again and asserts the withdrawal. It disables origin keep alive so group churn is deterministic, and waits after the traffic_ctl call because http_config_cb schedules the reconfigure a second out — without that wait the next request is still served by the previous HttpConfigParams, which looks exactly like a failure to retract.

MultiGroupAggregateTest gains an ExcludesExpression. Every assertion in that file was a ContainsExpression, which is why the original leak went unnoticed; a test that only checks for presence cannot catch a metric that should not be there.

Verified the autest fails without the fix: with the unlist call disabled, exactly one assertion fails, the retraction one. Four autests pass in a Fedora 44 container on the CI image — the three per_server* tests and slow_post, which exercises the same constructor through connection.max enforcement.

Two follow-ups from #13616

Both raised in review there after it merged, so they land here.

Metrics.h used std::forward in for_each while getting <utility> only through another
include. It compiles today, but that is not a property this header controls, so the include is now
direct.

Metrics.h is installed, so removing Metrics::iterator, begin(), end(), find(),
createSpan() and rename() breaks downstream plugins even with no in-tree callers. The v11
section of doc/release-notes/upgrading.en.rst now records the removals and shows the for_each
replacement, which is where an upgrader looks.

[Copilot]

🟡 Changes recommended

Moderate issues remain in the build configuration, metric ownership handling, and retraction test setup.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates per-server connection metrics to support runtime aggregate-mode changes and retract obsolete metric names.

Changes:

• Adds aggregate modes and metric retraction.
• Adds unit and end-to-end coverage.
• Updates configuration, documentation, and Metrics API migration notes.

File summaries

File	Reviewed changes and findings
tests/gold_tests/origin_connection/per_server_connection_max.test.py	Adds retraction coverage. Moderate (1 vote): retain ATS, origin, and DNS processes across runs. Nit (1 vote): rename the obsolete AggregateOnlyWithoutHostAggregateTest. Nit (3 votes): update the assertion message from AGGREGATE_ONLY to AGGREGATE_SUM.
src/records/RecordsConfig.cc	Allows aggregate values 0–3. No findings.
src/iocore/net/unit_tests/test_ConnectionTracker.cc	Tests aggregate modes and transitions. No findings.
src/iocore/net/ConnectionTracker.cc	Implements mode-specific publication and unlisting. Moderate (2 votes): avoid unlisting names still owned by live MATCH_HOST metrics in mixed groups.
src/iocore/net/CMakeLists.txt	Registers the connection-tracker tests. Moderate (3 votes): remove the duplicate source entry in test_net.
include/tsutil/Metrics.h	Adds the direct <utility> dependency. No findings.
include/iocore/net/ConnectionTracker.h	Documents aggregate modes and lifecycle. No findings.
doc/release-notes/upgrading.en.rst	Documents removed Metrics APIs. No findings.
doc/admin-guide/monitoring/statistics/core/http-connection.en.rst	Updates metric documentation. Nit (1 vote): refer to aggregate outputs as metrics, or explicitly identify the counters.
doc/admin-guide/files/records.yaml.en.rst	Documents dynamic aggregation behavior. No findings.

Review details

Suppressed comments (3)

doc/admin-guide/monitoring/statistics/core/http-connection.en.rst:238

• current_connection is documented as a Gauge immediately above, so calling all three aggregate outputs “counters” is inaccurate. Please refer to them as metrics (or name the two counters explicitly).

For a hostname aggregate there are two kinds. The *sums* are those same three counters, each added
across the groups of that hostname which have aggregation enabled, published at

tests/gold_tests/origin_connection/per_server_connection_max.test.py:528

• This test now exercises AGGREGATE_MAX, but its class and invocation remain AggregateOnlyWithoutHostAggregateTest, referring to the removed AGGREGATE_ONLY mode. Rename it (for example, AggregateMaxWithoutHostAggregateTest) so test output and future references describe the behavior being covered.

    metric_aggregate 2 (AGGREGATE_MAX) normally leaves the per group metrics hidden and publishes

tests/gold_tests/origin_connection/per_server_connection_max.test.py:642

• This helper retains only ATS, but the first run also starts self._server and the shared DNS. The later test runs issue more curls without starting or retaining those processes, so the origin/DNS are not guaranteed to remain alive and the retraction check can fail before exercising metric publication. Keep both long-lived dependencies in StillRunningAfter for every run (and call _use_shared_dns on the later runs).

    def _curl(self, tr) -> None:
        """Drive one request through the remap rule."""
        tr.MakeCurlCommand(f"-v --fail -s -x 127.0.0.1:{self._ts.Variables.port} 'http://retract.origin.com/get'", ts=self._ts)
        tr.Processes.Default.ReturnCode = 0
        tr.StillRunningAfter = self._ts

• Files reviewed: 10/10 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[bneradt]

[P2] Preserve metrics still published by a MATCH_HOST group

Confirmed by tracing the constructor and name generation. This does not require conflicting aggregate settings: keep a MATCH_HOST connection for example.test alive with metric_enabled=1 and metric_aggregate=0, then construct a MATCH_BOTH group for the same hostname with those same metric settings. The first group publishes current_connection.example.test, total_connection.example.test, and blocked_connection.example.test; the second reaches this branch and unlists all three. No hostname aggregate ever needed retracting in this sequence. Subsequent connections reusing the live MATCH_HOST group do not run its constructor, so its metrics remain absent from enumeration despite continuing to update.

Please distinguish aggregate publication from the MATCH_HOST group's publication requirement before withdrawing a shared name. Add coverage that keeps the MATCH_HOST group reserved while constructing a MATCH_BOTH group at mode 0 (and mode 2), then verifies all three MATCH_HOST metrics remain listed. The current fallback coverage only tests MATCH_PORT, which cannot expose this name collision.

[bneradt]

Reviewed 3854c19. One correctness issue needs addressing before approval: the new sum-retraction branch can withdraw a live MATCH_HOST group's own metrics when a MATCH_BOTH group for the same hostname is constructed, even when both use metric_aggregate=0. I added the concrete sequence and regression-test request to the existing inline discussion: #13666 (comment).

The four-mode publication logic otherwise looks consistent for MATCH_BOTH groups, and the Metrics include and v11 migration documentation follow-ups are addressed. The duplicate test_ConnectionTracker.cc CMake entry is worth removing as already requested, but I am not treating it as a separate correctness blocker.

Validation: source/test review and clean git diff --check. Thirteen CI checks pass; Clang-Analyzer is still pending. I did not run a local build or tests.