perp-fees: delete gauges on asset_not_found (Lighter ghost metrics) - #1619
Closed
Flotapponnier wants to merge 7 commits into
Closed
perp-fees: delete gauges on asset_not_found (Lighter ghost metrics)#1619Flotapponnier wants to merge 7 commits into
Flotapponnier wants to merge 7 commits into
Conversation
* bridge-monitor: add Squid, Socket, Mayan bridges + bench specs * feat: Squid/Socket/Mayan bridges + per-region quote latency + corridor p50/p99 matrix
cron: warm /perps hub in health-check every 5 min
- ethereum-rpc answer: 8→7 providers, remove Flashbots (now on mev-protect-rpc), bench FAQ 15s→60s cadence - bridge-quote-latency answer+bench: add Squid/Socket (now 8 bridges), update single eu-west→3 regions - cheapest-bridge answer: add Squid/Socket (now 8 routers, was 6), remove false 'not measured' claim - staking answer: update 3→7 chains to match validator-yield bench, enumerate all 7 in short_answer and limitations - bridge page: add Squid/Socket to description and tooltip, Fragment key fix
Lighter removed BTC/ETH/SOL from their perp markets. The harness was returning asset_not_found on every tick but recordSample() returned early without deleting the gauge, so the Go Prom client kept the last set value in memory forever. Classifying asset_not_found as not_found and calling DeleteLabelValues clears all gauge series (allInBps, spread, takerFee, funding, tiers) so Lighter/BTC etc. drops off the rankings instead of carrying stale top-ranked values indefinitely.
Flotapponnier
commented
Jul 28, 2026
CollaboratorAuthor
Superseded by #1621 (clean branch from dev, no merge conflicts) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Lighter removed BTC, ETH, and SOL from their perp markets. Their live
/orderBookDetailsAPI returns 89 markets, none of which are BTC/ETH/SOL. The harness has been returningasset_not_foundon every 5-minute tick for all three assets.The bug:
recordSample()returned early on error without clearing the gauge. The Go Prometheus client retains the last.Set()value in memory indefinitely. So Lighter kept showing as #1 for BTC all-in cost (0.35 bps) — a ghost metric from before the delisting.This affected the headline ranking AND all notional tier gauges ($10k, $100k, $1M), which is why the
perp-fees?chain=BTC&view=all_in_100kpage showed a suspiciously great Lighter value.Fix
not_founderror class toclassifyErr()(matchesasset_not_foundandunsupported_venue)errType == "not_found", delete all gauge series for that venue/asset:allInBps,spread,takerFee,funding,fetchLatency,lastRefresh, and all four tier gaugesDeploy
After merging to dev → main, rebuild and redeploy the
perp-feesharness on the VPS so the running process picks up the fix and clears the stale Lighter/BTC metrics on its next tick.