feat: snapshot + carry-forward for /perps and /hyperliquid cohorts - #689
Merged
Conversation
Adds an Upstash-backed cohort snapshot layer so the two hub pages serve last-known-good data even when Prom blips. A Vercel cron writes a fresh JSON snapshot every minute; the readers (fetchPerpCohort, fetchHlCohort, fetchHlHip3Cohort) prefer the snapshot, fall through to live Prom on stale or missing blobs, and write back on success.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/lib/cohort-snapshot.ts(Upstash REST, single{data, asOf}envelope per key, 24 h TTL safety net, 10 min default staleness ceiling). Mirrorssrc/lib/materialize/store.tswithout the pointer indirection since cohort aggregates are a single blob.fetchPerpCohort,fetchHlCohort,fetchHlHip3Cohortnow read Upstash first, fall through to live Prom + writeback on miss/stale, and are wrapped inunstable_cache(60 s) for concurrent-request collapsing. The uncached Prom path is exported as*Freshfor the cron./api/cron/snapshot-perp-cohort(keyperp-cohort) and/api/cron/snapshot-hl-cohort(keyshl-frontends+hl-hip3, refreshed in parallel with per-key error isolation). Both gated by the existingCRON_SECRETBearer header, structure mirrorshealth-check/indexnow.vercel.jsongains the two cron entries.Why
Both hub pages currently fetch live from Prom on render. A single Prom hiccup (Railway redeploy, scrape miss, harness restart) parks a null cohort in Vercel ISR and the user sees "..." everywhere until the next refresh. With the snapshot layer, the worst case becomes a 5 min stale badge during a harness blip instead of an empty leaderboard.
Reused infra (no new env vars)
CRON_SECRET(already on Vercel, gateshealth-check/indexnow)KV_REST_API_URL+KV_REST_API_TOKEN(already used bysrc/lib/snapshot.tsper-bench snapshots; falls back toUPSTASH_REDIS_REST_URL/UPSTASH_REDIS_REST_TOKENto match the materialize layer)Shape check
The snapshot envelope wraps
PerpCohortSummary | HlCohortSummary | HlHip3Summaryverbatim; the reader returnsenvelope.dataof the same type, no field drift.Test plan
vercel.com/.../settings/cron-jobs)/api/cron/snapshot-perp-cohortmanually with the CRON_SECRET to seed the blob, then verify/perpsrenders cohort cards/api/cron/snapshot-hl-cohortand/hyperliquidPROMETHEUS_URLat a dead host on a preview) and confirm both hubs keep serving the last good snapshot until the 10 min staleness window