derivation: `baseHeight` default and the below-baseHeight batch skip are both silent, making a frozen `safe`/`finalized` undiagnosable

[panos-xyz]

Summary

Two pieces of derivation state are set silently, and together they can freeze the safe/finalized L2 block tags indefinitely with zero errors or warnings anywhere in the logs:

1. baseHeight defaults to the L2 head captured at process start (node/derivation/derivation.go:183-189) and is never logged.
2. The per-batch skip that this default causes (node/derivation/derivation.go:484-486) is a bare continue with no log line.

The result is a node that looks completely healthy — derivation running, L1 cursor advancing, eth_getLogs succeeding, batches being fetched and content-verified — while eth_getBlockByNumber("safe") and ("finalized") both return null forever.

Impact

Observed on a Morph mainnet fullnode running against morph-reth (2026-08-24 → 2026-08-27, ~3 days):

• derivation started, 583 × local verify fetched batch metadata, L1 cursor advancing steadily (500 blocks / 15 s), fetched rollup tx txNum=2..3 per poll, zero Error lines in the derivation module.
• eth_getBlockByNumber("safe") → null, ("finalized") → null, for the entire 3 days.
• Downstream, this made the execution client's ExEx WAL grow without bound: reth only prunes <datadir>/exex/wal from ExExManager::finalize_wal, which is driven exclusively by the finalized-header stream. With finalized never set, the WAL reached 2.6 GB of a 40 GB datadir (~28 KB per block × ~93 k blocks — exactly the 4 days of uptime). Since reth must replay the entire WAL before opening RPC, every restart got progressively slower.

The chain of causation is entirely invisible from the logs, so diagnosing it required reading the derivation source and eliminating every other exit path by contradiction.

Root cause

newDerivation:   baseHeight == 0  →  baseHeight = current L2 head          (derivation.go:183-189, NOT logged)
derivationBlock: lastHeader.Number <= baseHeight  →  continue             (derivation.go:484-486, NOT logged)
                 → tagAdvancer.advanceSafe() never called                 (derivation.go:510)
finalizerTick:   safeNum == 0  →  return                                  (finalizer.go:88-89)
                 → SetBlockTags never called with a non-zero hash
                 → EL never sets a finalized block  →  ExEx WAL never pruned

The trigger is a stale derivation cursor relative to the local L2 head: if node-data's persisted LatestDerivationL1Height is well behind L1 while the EL is already near the chain tip (e.g. an EL snapshot restored next to older node data, or a low --derivation.startHeight), then every batch derivation replays has a lastL2BlockNumber below the startup L2 head. 484 therefore fires for every batch, and the cursor still advances at the end of derivationBlock, so from the outside it looks like normal forward progress.

Note this is not a rare edge case: the further the derivation cursor lags the EL head, the longer the condition holds, and it holds for the entire catch-up window.

Why it is undiagnosable today

startHeight's first-run default is logged:

// derivation.go:174
logger.Info("derivation startHeight defaulted to latest L1 confirmed block", "height", blockNumber, "confirmations", d.confirmations)

baseHeight's first-run default, immediately below it, is not — even though it silently gates advanceSafe for every batch. And the continue it drives emits nothing at all, so there is no way to tell "skipped 583 batches because they are below baseHeight" apart from "processed 583 batches normally".

Proposed change

1. Log the resolved baseHeight at startup, mirroring the existing startHeight line, and state where the value came from (CLI/env vs. the L2-head default):

   // derivation.go, after the baseHeight block at 183-189
   logger.Info("derivation baseHeight resolved", "height", d.baseHeight, "source", baseHeightSource)

2. Make the batch skip observable. A rate-limited Info (per N skips, or at most one per poll/minute) carrying enough context to act on:

   // derivation.go:484
   if lastHeader.Number.Uint64() <= d.baseHeight {
       // rate-limited
       d.logger.Info("batch below baseHeight; skipping safe advance",
           "batchIndex", batchInfo.batchIndex,
           "lastBlockNumber", lastHeader.Number.Uint64(),
           "baseHeight", d.baseHeight,
           "skippedSinceStart", n)
       continue
   }

Optionally, expose baseHeight and a derivation_batches_skipped_below_base_height counter as metrics — with those two, the condition is visible on a dashboard rather than requiring a source read.

Both changes are logging/observability only; no behavioural change is intended. The current skip semantics are correct — the problem is purely that they are invisible.

Reproduction

1. Restore an EL datadir that is near the chain tip.
2. Pair it with node-data whose LatestDerivationL1Height is well behind L1 (or pass a low --derivation.startHeight).
3. Start the node without --derivation.baseHeight, in the default --derivation.verify-mode=local.
4. Observe: derivation logs steady forward progress with no errors, while eth_getBlockByNumber("safe") and ("finalized") stay null until the L1 cursor reaches batches whose lastL2BlockNumber exceeds the startup L2 head.

Environment

• morph-l2/morph main @ 2519cf1c
• Morph mainnet, fullnode (no signer, derivation enabled), --derivation.verify-mode=local (default)
• Execution client: morph-reth