Skip to content

fix(source-files): key document cache on content hash, not mtime - #94

Open
marcleblanc2 wants to merge 1 commit into
timlrx:mainfrom
marcleblanc2:fix/content-hash-cache-key
Open

marcleblanc2 wants to merge 1 commit into
timlrx:mainfrom
marcleblanc2:fix/content-hash-cache-key

Conversation

@marcleblanc2

Copy link
Copy Markdown

Problem

@contentlayer2/source-files keys its document cache on the file's mtime:

const documentHash = yield* $(pipe(fs.stat(fullFilePath), T.map((_) => _.mtime.getTime().toString())))

Git doesn't store mtimes, so every fresh clone (Vercel, GitHub Actions, Netlify, …) stamps all content files with the checkout time. Even when .contentlayer/.cache is persisted between builds, no stored documentHash ever matches, and every document is re-parsed and re-rendered on every CI build. The cache only ever hits on a working tree that persists between runs (a developer laptop).

We hit this on a ~500-document Next.js site: after wiring the cache dir into Vercel's build cache, the cache loaded fine but was then 100% ignored.

Fix

Read the file once, hash its contents with the existing hashObject (xxhash64 from @contentlayer2/utils, already used for the schema hash), and pass the contents to processRawContent so a miss doesn't read the file twice.

  • documentHash is now stable across clones and only changes when the content changes.
  • processRawContent no longer needs fs.HasFs; fs.StatError is no longer possible, HashError is mapped to UnexpectedError like the other IO errors.
  • generate-dotpkg also uses documentHash to skip unchanged .json writes, so it benefits from the same stability.
  • Updated the stale DataCache.CacheItem doc comment that still described the mtime scheme.
  • Changeset: patch for @contentlayer2/source-files and @contentlayer2/core.

Cache entries written by previous versions have mtime-shaped hashes and will miss once, then be rewritten with content hashes. Nothing else about the cache format changes.

Tests

Added to makeCacheItemFromFilePath.spec.ts (runTest gained an optional previousCache):

  • documentHash depends on file contents, not mtime — bumps the fixture's mtime by an hour with utimes and asserts the hash is unchanged, and that a file with different content gets a different hash. Fails on main (expected '1789625945676' to be '1789622345676'), passes with this change.
  • returns the previous cache item when the hash matches — marks the cached document and asserts the marker comes back.
  • re-parses the document when the hash differs — stale hash in previousCache → marker absent, fresh hash returned.

yarn build:ts, yarn lint:check, and yarn test (all workspaces) pass locally.

Also verified end to end with the built CLI on a scratch project: after touch -t 203001010000 content/*.md the persisted data-*.json hashes are byte-identical; editing one file changes only that file's hash.

Git doesn't preserve file mtimes, so on a fresh clone (CI, Vercel, ...)
every document's mtime differs from the one stored in the cache and the
whole content directory is re-rendered, even when .contentlayer/.cache
is persisted between builds.

Read the file once, hash its contents with the existing xxhash64
hashObject helper, and reuse the contents for parsing on a miss.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad94-33bf-70cc-a0cc-45aabccdcda7
Co-authored-by: Amp <amp@ampcode.com>
marcleblanc2 added a commit to sourcegraph/docs that referenced this pull request Sep 17, 2026
Every Vercel build spent 25-40s of its ~2 minutes re-rendering all 506
MDX documents, even when they didn't change, because contentlayer2's
cache lived in `.contentlayer/.cache` and Vercel only keeps
`.next/cache` between builds

## Changes

- `pnpm run build` now runs contentlayer2 as its own step
(`dev/build-content.mjs`) with its cache under `.next/cache`, so a
deploy re-renders only the documents that changed

- Upstream Contentlayer2 decides "changed" by comparing each file's
`mtime`, but Vercel builds use fresh clone, which sets every `mtime` to
the clone time, so the restored cache never hit
- `dev/build-content.mjs` now sets each `.mdx` file's `mtime` from a
hash of its content first
- Opened timlrx/contentlayer2#94 to fix this upstream; drop this
workaround once the fix is shipped

- This removes the only reason `next build` was still holding onto
webpack, so we switched it to using Vercel's Turbopack for faster builds
and better caching

- This upgrade to Turbopack required 2 fixes in our code:
  - `src/data/redirects.ts` is imported instead of `require()`
- `contentlayer.config.ts` imports the Shiki theme as JSON instead of by
a relative path

## Vercel build times

| Step | Cold build on `main` | Warm build on this branch |
| --- | --- | --- |
| contentlayer | ~40s | 3s |
| compile | ~19s | 4.7s |
| TypeScript | 7s | 4.5s |
| static pages | ~37s | ~27s (1512 pages since #1945) |
| build traces | ~8s | – |
| **Build Completed** | **2m** | **49s** |

- The first build after any change to this script is still cold (its
restored cache holds the old hashes)
  - Every build after is warm

- Also checked locally:
- Touching every `.mdx` without changing content still hits the cache
(8.5s → 2.8s)
  - Changing one file re-renders only that file
- `next start` serves docs pages, `/api/og`, `.md` rewrites, redirects,
sitemap, `/api/releases`, 404
  - Shiki output identical to webpack
  - `tsc`, lint and `check-redirects` unchanged from `main`

---------

Co-authored-by: Amp <amp@ampcode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant