fix(source-files): key document cache on content hash, not mtime - #94
Open
marcleblanc2 wants to merge 1 commit into
Open
marcleblanc2 wants to merge 1 commit into
marcleblanc2 wants to merge 1 commit into
Conversation
Git doesn't preserve file mtimes, so on a fresh clone (CI, Vercel, ...) every document's mtime differs from the one stored in the cache and the whole content directory is re-rendered, even when .contentlayer/.cache is persisted between builds. Read the file once, hash its contents with the existing xxhash64 hashObject helper, and reuse the contents for parsing on a miss. Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad94-33bf-70cc-a0cc-45aabccdcda7 Co-authored-by: Amp <amp@ampcode.com>
marcleblanc2
added a commit
to sourcegraph/docs
that referenced
this pull request
Sep 17, 2026
Every Vercel build spent 25-40s of its ~2 minutes re-rendering all 506 MDX documents, even when they didn't change, because contentlayer2's cache lived in `.contentlayer/.cache` and Vercel only keeps `.next/cache` between builds ## Changes - `pnpm run build` now runs contentlayer2 as its own step (`dev/build-content.mjs`) with its cache under `.next/cache`, so a deploy re-renders only the documents that changed - Upstream Contentlayer2 decides "changed" by comparing each file's `mtime`, but Vercel builds use fresh clone, which sets every `mtime` to the clone time, so the restored cache never hit - `dev/build-content.mjs` now sets each `.mdx` file's `mtime` from a hash of its content first - Opened timlrx/contentlayer2#94 to fix this upstream; drop this workaround once the fix is shipped - This removes the only reason `next build` was still holding onto webpack, so we switched it to using Vercel's Turbopack for faster builds and better caching - This upgrade to Turbopack required 2 fixes in our code: - `src/data/redirects.ts` is imported instead of `require()` - `contentlayer.config.ts` imports the Shiki theme as JSON instead of by a relative path ## Vercel build times | Step | Cold build on `main` | Warm build on this branch | | --- | --- | --- | | contentlayer | ~40s | 3s | | compile | ~19s | 4.7s | | TypeScript | 7s | 4.5s | | static pages | ~37s | ~27s (1512 pages since #1945) | | build traces | ~8s | – | | **Build Completed** | **2m** | **49s** | - The first build after any change to this script is still cold (its restored cache holds the old hashes) - Every build after is warm - Also checked locally: - Touching every `.mdx` without changing content still hits the cache (8.5s → 2.8s) - Changing one file re-renders only that file - `next start` serves docs pages, `/api/og`, `.md` rewrites, redirects, sitemap, `/api/releases`, 404 - Shiki output identical to webpack - `tsc`, lint and `check-redirects` unchanged from `main` --------- Co-authored-by: Amp <amp@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
@contentlayer2/source-fileskeys its document cache on the file's mtime:Git doesn't store mtimes, so every fresh clone (Vercel, GitHub Actions, Netlify, …) stamps all content files with the checkout time. Even when
.contentlayer/.cacheis persisted between builds, no storeddocumentHashever matches, and every document is re-parsed and re-rendered on every CI build. The cache only ever hits on a working tree that persists between runs (a developer laptop).We hit this on a ~500-document Next.js site: after wiring the cache dir into Vercel's build cache, the cache loaded fine but was then 100% ignored.
Fix
Read the file once, hash its contents with the existing
hashObject(xxhash64 from@contentlayer2/utils, already used for the schema hash), and pass the contents toprocessRawContentso a miss doesn't read the file twice.documentHashis now stable across clones and only changes when the content changes.processRawContentno longer needsfs.HasFs;fs.StatErroris no longer possible,HashErroris mapped toUnexpectedErrorlike the other IO errors.generate-dotpkgalso usesdocumentHashto skip unchanged.jsonwrites, so it benefits from the same stability.DataCache.CacheItemdoc comment that still described the mtime scheme.@contentlayer2/source-filesand@contentlayer2/core.Cache entries written by previous versions have mtime-shaped hashes and will miss once, then be rewritten with content hashes. Nothing else about the cache format changes.
Tests
Added to
makeCacheItemFromFilePath.spec.ts(runTestgained an optionalpreviousCache):documentHash depends on file contents, not mtime— bumps the fixture's mtime by an hour withutimesand asserts the hash is unchanged, and that a file with different content gets a different hash. Fails onmain(expected '1789625945676' to be '1789622345676'), passes with this change.returns the previous cache item when the hash matches— marks the cached document and asserts the marker comes back.re-parses the document when the hash differs— stale hash inpreviousCache→ marker absent, fresh hash returned.yarn build:ts,yarn lint:check, andyarn test(all workspaces) pass locally.Also verified end to end with the built CLI on a scratch project: after
touch -t 203001010000 content/*.mdthe persisteddata-*.jsonhashes are byte-identical; editing one file changes only that file's hash.