Retract per server metrics when the mode changes

doc/admin-guide/files/records.yaml.en.rst

@@ -2033,13 +2033,15 @@ Origin Server Connect Attempts
    this setting resolve to the same group -- that is, the same key under
    :ts:cv:`proxy.config.http.per_server.connection.match` -- the transaction that creates the group
    determines its metrics, and later transactions do not change them. A group is discarded once its
-   connection count reaches zero, so *raising* the level of publication is picked up the next time
-   that upstream is reopened: enabling metrics, or enabling the aggregates, takes effect as upstreams
-   reconnect. Lowering it does not. Metrics are never retired once published, so disabling this
-   setting, or switching
-   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` to ``2``, leaves the names that
-   are already published in place, frozen at their last sampled value, until |TS| is restarted. This
-   affects only which metrics exist; enforcement of
+   connection count reaches zero, so a change is picked up the next time that upstream is reopened.
+   A group that never goes idle keeps whatever was in effect when it was created.
+
+   Disabling this setting does not retire metrics that are already published: they stay in place,
+   frozen at their last sampled value, until |TS| is restarted. Changing
+   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` does retract what it no longer
+   asks for, as each group is rebuilt.
+
+   This affects only which metrics exist; enforcement of
    :ts:cv:`proxy.config.http.per_server.connection.max` uses the group's own connection count and is
    unaffected.
 
@@ -2051,24 +2053,30 @@ Origin Server Connect Attempts
    :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Has no effect when that setting
    is ``0``.
 
-   A per hostname aggregate sums a counter across every group belonging to that hostname that has
-   aggregation enabled, and exists only for
+   There are two kinds of per hostname aggregate. The *sums* add ``current_connection``,
+   ``total_connection`` and ``blocked_connection`` across every group belonging to that hostname
+   that has aggregation enabled. The *max* is ``current_connection.max``, the largest
+   ``current_connection`` among those groups, which is the one that answers how close the busiest
+   group is to :ts:cv:`proxy.config.http.per_server.connection.max`. Both exist only for
    :ts:cv:`match type <proxy.config.http.per_server.connection.match>` ``both``, since that is the
    only match type whose group key carries the hostname. See :ref:`per-server-connection-metrics`.
 
-   ===== ======================================================================================
-   Value Effect
-   ===== ======================================================================================
-   ``0`` No aggregates. The per group metrics are published under their own names.
-   ``1`` Publish the per hostname aggregates and the per group metrics.
-   ``2`` Publish only the per hostname aggregates. The per group metrics from which they are
-         computed are collected but not published, which keeps the number of published metrics
-         proportional to hostnames rather than to groups.
-   ===== ======================================================================================
+   ===== =========== ====== =====
+   Value Per group   Sums   Max
+   ===== =========== ====== =====
+   ``0`` published   no     no
+   ``1`` published   yes    yes
+   ``2`` hidden      no     yes
+   ``3`` hidden      yes    yes
+   ===== =========== ====== =====
+
+   ``2`` is the smallest useful configuration: one metric per hostname. ``3`` adds that hostname's
+   totals. Both keep the number of published metrics proportional to hostnames rather than to
+   groups.
 
-   With value ``2``, a group that has no aggregate to belong to -- any match type other than
-   ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would be
-   reported for it.
+   With values ``2`` and ``3``, a group that has no aggregate to belong to -- any match type other
+   than ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would
+   be reported for it.
 
    Values ``0`` and ``1`` can produce a very large number of metrics when the match type includes the
    address or port, since there is then one set per address and port rather than one per hostname.
@@ -2079,12 +2087,17 @@ Origin Server Connect Attempts
    upstream had aggregation enabled, so mappings that disagree for one hostname produce an aggregate
    that covers only part of it.
 
-   The reload is one-directional for the same reason given under
-   :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Raising the value takes effect
-   as upstreams reconnect, but moving to ``2`` does not hide per group metrics that are already
-   published, and moving from ``1`` to ``0`` does not stop the hostname aggregates from publishing.
-   Reducing the number of published metrics therefore requires a restart, which matters most for
-   ``2``, the value chosen specifically to bound that number.
+   A change in either direction takes effect as upstreams reconnect: a group publishes what the
+   new value asks for and withdraws what it does not, when that group is next rebuilt. Metrics
+   withdrawn this way stop appearing in :program:`traffic_ctl` output and in the other metric
+   consumers; they are not destroyed, and moving back republishes them with their accumulated
+   values intact.
+
+   The per group metrics belong to a single group, so raising the value withdraws them as that group
+   is rebuilt. The sums and the max are named per hostname and shared by its groups, so a group
+   rebuilt for a value that does not publish them only stops contributing; they are withdrawn once
+   no group of that hostname publishes them. Mappings that disagree for one hostname therefore
+   cannot hide each other's aggregate.
 
 .. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_prefix STRING NULL
    :reloadable:

doc/admin-guide/monitoring/statistics/core/http-connection.en.rst

@@ -234,21 +234,23 @@ blocked_connection
    Counter. The total number of connection attempts to the group blocked by
    :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
 
-For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
-hostname which have aggregation enabled, plus:
+For a hostname aggregate there are two kinds. The *sums* are those same three counters, each added
+across the groups of that hostname which have aggregation enabled, published at
+:ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` ``1`` and
+``3``. The *max*, published at ``1``, ``2`` and ``3``, is:
 
-current_connection_max
+current_connection.max
    Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment
    of sampling, so the maximum rather than the sum of the groups' current counts. This is useful
    because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than
    per hostname, so the busiest group is what determines whether connections are about to be
    blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water
-   mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate.
+   mark. There is no per group ``current_connection.max``; it exists only as a hostname aggregate.
 
 Because :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` is overridable, a group
 joins its hostname's aggregate only if the mapping that first opened that upstream had aggregation
 enabled. Mappings that disagree for one hostname therefore produce an aggregate over part of it: the
-sums cover a subset of the groups and ``current_connection_max`` takes its maximum over that same
+sums cover a subset of the groups and ``current_connection.max`` takes its maximum over that same
 subset, with nothing in the metric to indicate it. Keeping the setting uniform across the mappings
 for a hostname avoids this.
 
@@ -261,13 +263,13 @@ Every published per server metric is recomputed periodically, currently every 5
 on every connection event, so a reader sees a value up to that interval old. This is true of the
 hostname aggregates and of the published per group metrics alike: those are
 mirrored from the internal ones by the same periodic mechanism, not written as connections open and
-close. It applies to ``current_connection_max`` too, which reports the maximum across groups as of
+close. It applies to ``current_connection.max`` too, which reports the maximum across groups as of
 the last sample rather than a running peak. To obtain the peak over a longer window, compute a
 maximum over time from this gauge in the monitoring system.
 
-At :ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` value ``2``
-the per group metrics still exist internally, since the aggregates are computed from them, but are not
-published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which
+At :ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` values
+``2`` and ``3`` the per group metrics still exist internally, since the aggregates are computed from
+them, but are not published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which
 reads them directly and so is not subject to the sampling delay above. That visibility is intended
 for debugging and is not a stable interface: the existence, granularity and naming of the per group
 metrics may change independently of the published aggregates.

doc/developer-guide/internal-libraries/Metrics.en.rst

@@ -171,6 +171,18 @@ Adding a source that is already registered for that derived metric is a no-op, s
 re-register the same source, such as one recreating an object for the same key, need not track that
 itself. The ``type`` and ``op`` arguments are ignored if the derived metric already exists.
 
+``ts::Metrics::Derived::remove_source()`` is the counterpart, for a contributor that goes away or
+stops wanting the aggregate published:
+
+.. code-block:: cpp
+
+    ts::Metrics::Derived::remove_source("proxy.process.example.total", per_thing_metric);
+
+A derived metric is shared by its sources, so this does not unlist it while any remain. Removing the
+last source leaves nothing to report, so the name is unlisted; adding a source again relists it.
+Removing a source that is not registered, or naming a derived metric that does not exist, is a
+no-op.
+
 A hidden source can feed a published aggregate:
 
 .. code-block:: cpp

doc/release-notes/upgrading.en.rst

@@ -30,6 +30,25 @@ with :cpp:func:`TSPortDescriptorDestroy`. The descriptor can be destroyed
 immediately after :cpp:func:`TSPortDescriptorAccept` returns because the
 listener does not retain it.
 
+``ts::Metrics``, in the installed ``tsutil/Metrics.h``, no longer has an
+iterator. ``Metrics::iterator``, ``begin()``, ``end()`` and ``find()`` are
+removed, and enumeration is now ``Metrics::for_each(func)``, which invokes
+``func(name, type, value)`` for each metric:
+
+.. code-block:: cpp
+
+    ts::Metrics::instance().for_each([](std::string_view name, ts::Metrics::MetricType type, int64_t value) {
+      // ...
+    });
+
+Handing out a position let a caller name a slot the store was free to change
+underneath them, which is what the iterator could not be made safe against.
+Reaching a single metric by name is ``lookup()``.
+
+``Metrics::Storage::createSpan()`` and ``Metrics::rename()`` are also removed.
+Spans handed out unnamed slots that only ``rename()`` could name, and
+``rename()`` mutated a name that the lock free readers hand out views of.
+
 Upgrading to ATS v10.x
 ======================
 

include/iocore/net/ConnectionTracker.h

@@ -79,29 +79,54 @@ class ConnectionTracker
    *
    * This is independent of @c TxnConfig::metric_enabled, which decides only whether per server
    * metrics exist for a group at all. The per group metrics are always created in the hidden metric
-   * store; what varies here is what gets published from them:
-   *   - @c AGGREGATE_NONE: no aggregate. The per group metrics are published under their own names.
-   *     This is the default and matches the behavior of releases that had no aggregate support.
-   *   - @c AGGREGATE_GROUP: the per hostname aggregates are published, and so are the per group
-   *     metrics they are computed from.
-   *   - @c AGGREGATE_ONLY: the per hostname aggregates are published and the per group metrics stay
-   *     hidden, which keeps the published metric count proportional to hostnames rather than to
-   *     groups. Where a group has no aggregate to belong to -- see @c Group::host_metric_name, which
-   *     only yields a name for match type @c MATCH_BOTH -- the per group metrics are published
-   *     anyway, since otherwise nothing at all would be reported for that group.
+   * store; what varies here is what gets published from them.
+   *
+   * Two kinds of per hostname aggregate exist. The *sums* are @c current_connection,
+   * @c total_connection and @c blocked_connection added across the groups of a hostname. The *max*
+   * is @c current_connection.max, the largest @c current_connection among those groups. Which of
+   * them are published, and whether the per group metrics are published alongside, is what this
+   * selects:
+   *
+   *   | value                | per group | sums | max |
+   *   |----------------------|-----------|------|-----|
+   *   | @c AGGREGATE_NONE    | yes       | no   | no  |
+   *   | @c AGGREGATE_GROUP   | yes       | yes  | yes |
+   *   | @c AGGREGATE_MAX     | no        | no   | yes |
+   *   | @c AGGREGATE_SUM     | no        | yes  | yes |
+   *
+   * @c AGGREGATE_NONE is the default and matches the behavior of releases that had no aggregate
+   * support. @c AGGREGATE_MAX is the smallest useful configuration: one metric per hostname,
+   * answering how close the busiest group is to @c per_server.connection.max. @c AGGREGATE_SUM adds
+   * the totals for that hostname. Both keep the published metric count proportional to hostnames
+   * rather than to groups.
+   *
+   * Where a group has no aggregate to belong to -- see @c Group::host_metric_name, which only
+   * yields a name for match type @c MATCH_BOTH -- the per group metrics are published whatever this
+   * says, since otherwise nothing at all would be reported for that group.
    *
    * Keeping the per group metrics in the hidden store in every case means changing this at runtime
    * is only a change of what is registered for publication, with no metric to migrate between the
    * two stores.
    *
-   * The records layer validates and clamps this to 0..2. A plugin setting the overridable config
+   * A change is applied per group, when that group is next constructed, which happens on the first
+   * connection after its count last fell to zero. A group that never goes idle keeps whatever was
+   * in effect when it was created.
+   *
+   * The per group names belong to one group, so a group that stops publishing them retracts them,
+   * see @c ts::Metrics::unlist. The sums and the max are named per hostname and shared by its
+   * groups, so a group that stops publishing those only stops contributing to them, see
+   * @c ts::Metrics::Derived::remove_source; they are retracted once no group of that hostname
+   * publishes them.
+   *
+   * The records layer validates and clamps this to 0..3. A plugin setting the overridable config
    * directly is not clamped, see @c METRIC_AGGREGATE_CONV; any other value behaves as
-   * @c AGGREGATE_GROUP, publishing both the aggregate and the per group metrics.
+   * @c AGGREGATE_GROUP, publishing everything.
    */
   enum MetricAggregate : int {
     AGGREGATE_NONE  = 0, ///< No hostname aggregate; the per group metrics are published.
-    AGGREGATE_GROUP = 1, ///< Hostname aggregates published, along with the per group metrics.
-    AGGREGATE_ONLY  = 2, ///< Hostname aggregates published, per group metrics kept hidden.
+    AGGREGATE_GROUP = 1, ///< Hostname sums and max, published along with the per group metrics.
+    AGGREGATE_MAX   = 2, ///< Hostname max only; the per group metrics stay hidden.
+    AGGREGATE_SUM   = 3, ///< Hostname sums and max; the per group metrics stay hidden.
   };
 
   /// Per transaction configuration values.

include/tsutil/Metrics.h

@@ -33,6 +33,7 @@
 #include <limits>
 #include <string>
 #include <string_view>
+#include <utility>
 #include <variant>
 #include <optional>
 
@@ -718,6 +719,18 @@ class Metrics
      */
     static void add_source(std::string_view derived_name, Metrics::MetricType type, Metrics::AtomicType *source, Op op = Op::SUM);
 
+    /** Stop @a source contributing to a derived metric.
+     *
+     * The counterpart to @c add_source, for a contributor that goes away or stops wanting the
+     * aggregate published. A derived metric is shared by its sources, so this does not unlist it
+     * while any remain; when the last one is removed there is nothing left to report and the name
+     * is unlisted. Re-adding a source relists it.
+     *
+     * A source that is not registered for @a derived_name, or a name with no derived metric, is a
+     * no-op.
+     */
+    static void remove_source(std::string_view derived_name, Metrics::AtomicType *source);
+
     /**
      * Update derived metrics.
      *