Ingestion cron and tiered polling schedule

[vdvman1]

Question

How does the ingestion cron work, and what is the tiered polling schedule?

Settled going in: ingestion is tiered — the manifest plus the latest article polled often, older articles rarely or never. If-Modified-Since short-circuits re-fetches (Mojang's launcher API is believed to support it — verify). A content hash detects real change. An admin can force a refetch of a specific article. Backfill runs until all history is stored, then the cutover happens. Old articles are believed never to change, but the machinery must handle it if they do.

Decide:

• Cron topology. How many cron jobs, at what cadences, doing what. Cadence numbers come from the free-tier research.
• The tiers. What exactly counts as "latest" (the newest entry? every entry since the last release? anything under N days old?) and what the cold tier's cadence is — genuinely never, or a slow sweep.
• Change detection. If-Modified-Since first, hash second. Confirm Mojang's API actually honours conditional requests; if not, the hash carries the whole load. What is hashed — the raw body, so a processorVersion change is not mistaken for a content change.
• New version discovery. The manifest is the source of truth for which versions exist. What happens when a version disappears from the manifest?
• Backfill. One-off script or a cron tier that walks backwards? How is completion detected, and how do we know the corpus is whole before cutover?
• Failure handling. Mojang returns a 500, or a body that fails Zod parsing. What is stored, what is retried, what alerts. Parse failures are handled here and must never reach a rendered page — that is the premise the ZodError research is testing.
• Admin manual refetch — how it is triggered and who can trigger it. This is the escape hatch for a stale old article.
• Gap alerting — the map rejected a request-time fallback, so the cron must be loud when it falls behind rather than silently letting pages 404.
• What ingestion invalidates. Ingesting a new revision must invalidate the article's Next cache entry and trigger re-anchoring. Name the trigger; the rendering ticket owns the mechanism.

Blocked on the free-tier research and the article schema.

[vdvman1]

Constraint added from #33 (Convex preview deployments).

Preview deployments get their own crons. Without a guard, every open PR's preview would independently backfill all 403 articles from Mojang and keep polling for the 5-day preview TTL — the single largest free-tier cost leak found in that research.

The ingestion design must include a preview kill-switch (a Convex preview default environment variable is the suggested mechanism). Treat this as a requirement of this ticket, not separate wayfinding.

Full write-up: docs/research/convex-preview-deployments.md

[vdvman1]

Constraints from #32 (Convex free-tier research). Full write-up: docs/research/convex-free-tier-crons.md

Recommended cadence, with the arithmetic: crons.interval("poll-manifest", { minutes: 5 }) plus crons.daily("reconcile-articles", { hourUTC: 7 }). Over a 30-day month that is 8,640 + 30 ticks:

	usage	of free tier
Function calls	25,920 + 150	2.6% of 1,000,000
Action compute	0.075 + 0.031 GB-h	0.5% of 20
Database I/O	4.3 + 3.6 MB	~1% of 1 GB
Data egress	4.3 + 6.0 MB	~1% of 1 GB
Storage after backfill	18.5 MB	3.6% of 512 MB

5 minutes is set by Mojang, not by cost. Their CDN sends Cache-Control: max-age=240, so polling faster than ~4 minutes cannot see fresher data. 1-minute polling would still fit the free tier (13% of calls) but would buy nothing.

Three findings that change this ticket's design:

1. Drop If-Modified-Since — it does not work here. Measured directly: a 2018 article and a 2026 article return the same Last-Modified: Mon, 10 Aug 2026 16:02:36 GMT, because Mojang re-uploads the whole container on every publish. Conditional requests cannot distinguish a changed article from an unchanged one. Content hashing carries the entire load. The map's standing decision has been corrected accordingly.
2. Outbound fetch response bodies are not metered. Confirmed in convex-backend source — track_fetch_egress(origin, request_size), commented "Only track egress - because AWS only charges egress". The pessimistic reading would have made a 5-minute poll cost 2.16 GB/month (216% of allowance); it is actually 4.3 MB. This is what makes frequent polling affordable at all.
3. Convex bills whole documents on read. The reconcile sweep must read narrow hash documents, never full article documents — 3.6 MB/month versus 1.35 GB/month. This is a joint constraint with Article and revision schema in Convex #37.

Also: the free plan is a hard stop, never billed (the deployment returns HTTP errors over quota), and quotas are per team — dev, preview and production share one pool. Combined with #33's finding that preview deployments run their own crons, the preview kill-switch already noted on this ticket is not optional.