Article and revision schema in Convex

[vdvman1]

Question

What is the Convex schema for articles and their revisions?

Settled going in (see the map's standing decisions): articles are stored fully processed — sanitized, list-nesting fixed, headings id'd, syntax-highlighted, plus the section tree and extracted plain text. Every revision is kept indefinitely. Storage is the architecture; revision browsing is not.

Decide:

• Table shape. One articles table with a pointer to the current revision plus a revisions table? Or revisions only, with latest derived? What is keyed by version id vs revision id?
• What a revision holds — processed HTML, section tree, plain text, content hash, fetched-at, the raw body as fetched (do we keep it, for reprocessing without re-fetching Mojang?).
• What lives on the article rather than the revision — the manifest entry fields (title, type, image, date, shortText, contentPath). Note that shortText currently gets a regex spacing fix applied at parse time (src/server/java/versions.ts); decide whether that stays a parse-time transform.
• processorVersion. Where it is stored, how a re-process pass finds stale revisions, whether re-processing creates a new revision or mutates in place. This matters: a content change and a processor change are different events and annotations must be able to tell them apart.
• Content hashing. What exactly is hashed — the raw fetched body, so a processor change does not look like a content change.
• Indexes, including the full-text search index. Constrained by the Convex FTS research.
• Manifest/version list storage — the sidebar needs the ordered version list; is that a query over articles or a separately maintained document?

Do not preclude the out-of-scope futures the map names: semantic search embeddings, Bedrock changelogs, mcmeta deep links (#6), and the mcdoc viewer (#5). Say explicitly what each would need and confirm the schema leaves room.

Use /domain-modeling — the vocabulary settled here (article, revision, section, anchor) is what every later ticket speaks.

[vdvman1]

Constraints added from #31 (Convex full-text search research). Full write-up: docs/research/convex-full-text-search.md

One question this ticket must not defer: article-level or section-level search rows? Section-level gives deep links straight to a heading and keeps each indexed document small; article-level is simpler but makes a hit mean "somewhere in this 15 KB page". This cannot be changed later without a reindex and a different result UI, so decide it here.

Recommended structural hedge: put the search index on a lean projection table (ids, type, a coarse date bucket such as releaseYear, and a single concatenated haystack field) rather than on the fat article document. Two reasons: candidate documents stay small against Convex's 16 MiB transaction read budget, and that projection is literally the document you would ship to Typesense/Meilisearch if the engine is ever outgrown. It costs nothing now and preserves the escape hatch the map already assumes.

Index only the current revision, never historical revisions — otherwise every edit multiplies the index and old text competes with new for the 1024-result ceiling.

Engine limits the schema has to design around:

• 1024 results, hard. MAX_CANDIDATE_REVISIONS = 1024 is passed as max_results on every query; pagination slices that fixed list. Paging past it throws rather than reporting exhaustion.
• Multi-term is OR, not AND — not configurable.
• No stemming, stop words, phrase queries, or negation. The pipeline is SimpleTokenizer → RemoveLongFilter(32) → LowerCaser. Fuzzy was deprecated in Jan 2025.
• No range filters, and post-.filter() date ranges are silently wrong — the top 1024 candidates are selected before your filter runs. Equality filters (e.g. version type) are fine, which is why a coarse bucket field beats a raw date.
• 1.21.4 tokenizes to 1, 21, 4. Version lookup must be a direct index lookup, never a search query. (24w45a survives tokenization.)
• No ordering control — relevance only, and Convex notes relevance order may change.

Cost is not a constraint: metering charges the whole index per search (bytes_searched = index_size), but at ~5 MB that is ~600,000 free searches/month against the 3,000 query-GB free tier.

[vdvman1]

Constraints from #32 (Convex free-tier research). Full write-up: docs/research/convex-free-tier-crons.md

A hard structural requirement, not a preference: the article body must live in its own document. Convex bills whole documents on read — there is no column projection. A daily reconcile sweep over narrow hash documents costs ~3.6 MB/month; the same sweep over fat article documents costs 1.35 GB/month against a 1 GB allowance — fatal. Split the narrow metadata/hash record from the heavy processed-HTML payload.

processorVersion bumps must overwrite in place, not create new revisions. Five bumps that each mint a revision would consume the entire storage reserve. This directly answers one of the open questions in this ticket's body: a reprocess is not a new revision. Content change and processor change are different events, and only the former creates a revision.

Measured sizing: the v2 manifest holds 403 articles (not the 800–1000 assumed when this map was charted). At ~46 KB processed per article, a full backfill is 18.5 MB of the 512 MB free-tier cap (3.6%). Reserving half the cap leaves headroom for roughly 5.6 revisions per article at 1,000 articles.

Free plan is a hard stop, never billed — over quota, the deployment returns HTTP errors. And quotas are per team, so dev, preview and production share one pool. That compounds with #33's finding about preview deployments running their own crons.

[vdvman1]

Constraints from #36 (anchoring prior art). Full write-up: docs/research/annotation-anchoring-prior-art.md

Three requirements this ticket must satisfy for anchoring to work at all:

1. Two version counters, not one. Split textVersion (text normalisation only) from processorVersion (rendering). Quotes and section paths then survive every rendering bump without a re-anchor pass — a significant saving given Convex free-tier limits and cron constraints #32's finding that reprocesses must overwrite in place.
2. The section tree needs character offsets. It currently carries none (see ArticleSection/ArticleSubSection in src/components/java/patch-notes.tsx), and without them a quote match cannot be scoped to a section. Anchoring depends on this.
3. Extract plain text from the sanitised source, not the rendered output. The rendered output is the thing that changes across processorVersion bumps; the sanitised source is stable. This also matters for the search projection (What can Convex built-in full-text search actually do? #31) — same field, same reasoning.

Context for why the structural leg cannot lean on the rendered DOM: measurement across all 403 articles found 42% with unbalanced <li> and 44% with unbalanced <p>, so the parsed tree is partly an error-recovery artefact and two parser versions may legitimately disagree.