Fail closed when shadow policy is enabled without shadow observability

[lan17]

Summary

Make positive shadow policy visibly fail closed when the configured metrics adapter cannot record shadow outcomes.

Today, scheduleShadowValidation() silently returns when metrics.shadowValidation is absent. A use case can therefore resolve a positive shadow.ramp while performing no shadow read, comparison, fill, or outcome observation, with no indication that the rollout policy is ineffective.

The intended behavior should remain: do not run detached shadow traffic invisibly. The improvement is to surface the configuration error rather than silently treating the policy as disabled.

Current behavior

Shadow work requires all of the following:

• valid remote policy;
• positive shadow ramp;
• deterministic ramp admission;
• available per-instance shadow capacity; and
• metrics.shadowValidation to exist.

The observability hook is currently checked after shadow policy validation, and its absence causes an unconditional return. This couples behavior to metrics wiring without a diagnostic.

Problem

A deployment can:

1. configure shadow.ramp > 0 in a static default or runtime provider;
2. omit the optional shadowValidation metrics hook, use an older custom adapter, or accidentally wire the wrong adapter;
3. observe no shadow traffic and no shadow outcomes; and
4. incorrectly assume that Redis is being validated or warmed before serving ramp increases.

This is especially risky because shadow mode exists specifically as a controlled rollout and validation mechanism. Silent non-execution defeats that purpose.

Proposed behavior

Static/default policy

When cached() registration or getOrLoad() invocation receives a positive effective static shadow.ramp and the instance metrics adapter has no shadowValidation hook, reject the configuration clearly before any shadow-capable cache execution begins.

Possible error:

DialCache shadow.ramp requires metrics.shadowValidation observability

For cached(), fail at registration. For getOrLoad(), fail during its normal enabled-path option/policy validation. Calls outside an enabled scope should retain true pass-through behavior and should not validate cache policy solely to reject shadow configuration.

If throwing for static getOrLoad() policy conflicts with established fail-open config semantics, the implementation may use the runtime behavior below consistently instead; the decision must be explicit and tested.

Runtime-provider policy

A runtime provider can enable shadow after registration. For a positive resolved runtime shadow ramp without metrics.shadowValidation:

• disable shadow work for that invocation;
• preserve normal caller cache behavior;
• record one bounded config_resolution error when an error hook exists;
• emit a safe warning that identifies only bounded definition metadata such as namespace, use case, and key type;
• avoid warning once per invocation through per-use-case deduplication or rate limiting.

Do not synthesize shadow outcomes when no shadow job was admitted.

Zero or omitted policy

shadow omitted, shadow.ramp omitted, or shadow.ramp: 0 remains inert and requires no shadow observer.

Why fail closed

Running detached source/Redis traffic without verdict observability is operationally unsafe:

• mismatch rates cannot be measured;
• fill/error/timeout/drop outcomes are invisible;
• operators cannot know whether a serving-ramp increase is justified;
• background load can exist without its intended control signal.

Disabling only shadow work preserves application correctness and the source-of-truth fallback while preventing invisible rollout traffic.

Alternative considered

Run shadow work even without shadowValidation and rely on logging or other metrics.

Rejected: the primary bounded outcome series is part of the shadow feature's safety contract. Logging every result is unsuitable, and generic request/error metrics cannot represent match, mismatch, superseded, filled, timeout, or dropped outcomes.

Compatibility

This changes only misconfigured positive shadow policy:

• configurations with shadow omitted or zero are unchanged;
• configurations with a working shadowValidation hook are unchanged;
• configurations that currently request shadow without the hook move from silent no-op to an explicit configuration failure/diagnostic;
• normal cache serving remains fail open where runtime policy is malformed or unsupported.

Because positive policy that previously did nothing may now reject static registration or emit new diagnostics, document the behavior change in release notes.

Acceptance criteria

• Positive static/default shadow policy without metrics.shadowValidation cannot remain a silent no-op.
• The chosen static behavior—registration rejection or explicit fail-closed diagnostic—is consistent across cached() and enabled getOrLoad() and is documented.
• Positive runtime-provider shadow policy without the hook disables only shadow work and preserves normal caller behavior.
• Runtime misconfiguration records bounded config_resolution observability where possible.
• Runtime warnings are deduplicated or rate-limited by bounded definition identity and cannot amplify an incident per invocation.
• No cache key, ID, source value, cached value, raw error, or other unbounded data is logged or labeled.
• Omitted and zero shadow policies remain inert without requiring the hook.
• A metrics adapter with shadowValidation continues to receive exactly one terminal outcome per admitted job.
• Missing observability does not consume shadow capacity, issue Redis commands, invoke the source, or schedule detached work.
• Tests cover static defaults, sparse runtime overlays, runtime enable/disable transitions, old custom metrics adapters, logger/metrics failures, and packed ESM/CommonJS consumers.
• README shadow-rollout guidance states that shadow outcomes are mandatory for positive shadow policy.

Non-goals

• Making every optional metrics hook mandatory.
• Adding a second shadow observer or callback.
• Running shadow work without verdict observability.
• Changing shadow ramp sampling, capacity, deadlines, comparisons, fills, or caller behavior.

Related

• Refactor cache config to group policy by layer #144 — layer-first config shape; positive remote.shadow.ramp should preserve this validation.
• Bound metrics error labels with a backend-neutral taxonomy #45 — bounded observability taxonomy.
• Consume rejected promises from logger and metrics adapters #103 — observer rejection isolation.