Skip to content

refactor: singular + enforced toolkit data contract - #1105

Merged
teallarson merged 14 commits into
mainfrom
chore/single-source-of-truth
Aug 5, 2026
Merged

refactor: singular + enforced toolkit data contract#1105
teallarson merged 14 commits into
mainfrom
chore/single-source-of-truth

Conversation

@teallarson

@teallarsonteallarson commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

This PR makes redirects and toolkit data use shared, typed sources instead of duplicated definitions and text parsing.

Before → after

AreaBeforeAfter
RedirectsData embedded in a 962-line next.config.ts; consumers parsed the file as text.Redirects live in redirects.ts; consumers import the data directly. The 156-entry array is unchanged.
Redirect checkingThe tested helper lived in an unused copy.The shipping checker imports the tested helper module.
Toolkit primitivesData-directory paths, ID normalization, categories, and slug logic were repeated across the app and generator.Shared primitives have one definition and all consumers use TOOLKIT_DATA_DIR consistently.
Data contractThe app used a small duck-check and then cast JSON to ToolkitData.The app and generator share Zod schemas; types are inferred from the same contract.
Invalid dataMissing files, invalid JSON, and invalid shapes were handled the same way and could silently hide a toolkit.Missing data returns null; invalid JSON or schema data throws with the file path and validation error.
Category safetyUnknown categories fell back to an unreachable others route.Missing categories produce non-clickable entries; unknown categories fail clearly; others is removed.
Toolkit readsRoute generation repeatedly scanned the toolkit directory.One cached loader indexes the directory once per process.

Compatibility

  • Redirect output is unchanged: 156 entries, same content and order.
  • Generated pages and routes remain unchanged.
  • The cache removes redundant directory passes; it is not presented as a measured build-time speedup.

Verification

  • pnpm test passes.
  • pnpm exec tsc --noEmit passes.
  • pnpm lint reports no errors and the same three pre-existing complexity warnings.

One intentional trade-off: a corrupt toolkit file now fails the directory load instead of affecting only that toolkit. In production, the build catches this before requests are served.


Note

Medium Risk
Changes affect static route generation, sitemap/redirect correctness, and production toolkit API reads—invalid JSON now fails builds/requests instead of hiding toolkits, which is safer but stricter.

Overview
Redirects move out of next.config.ts into importable redirects.ts (same rules, no behavior change). CI, pre-commit, check-redirects, update-links, and sitemap tests now use that module instead of regex-parsing config.

Toolkit docs data is a single contract: UI types are z.infer from generator toolkit-schemas; readToolkitFile / readToolkitIndex validate with Zod. Missing files still return null; bad JSON or schema mismatches throw with paths so builds break on corrupt nightly output. loadAllToolkitData + production lookup caching dedupe reads; slug vs normalized id cache keys stay distinct for the API route.

Routing / integrations: slug, category, and data-dir helpers live in toolkit-docs-generator/src/shared/; the fake others catch-all is gone—no category means toIntegrationLink returns null and cards are non-clickable; unknown categories throw. Sidebar sync and filters follow the same category list (design-system CATEGORIES for filters).

DX / ops: webpack extensionAlias for shared .js imports; zod is a runtime dependency; toolkit docs CI gets a Slack alert job on generation failure; new tests cover data dir, cache, parity, and category route dirs.

Reviewed by Cursor Bugbot for commit 53e7df1. Bugbot is set up for automated code reviews on this repo. Configure here.

teallarsonand others added 2 commits July 31, 2026 09:49
Deletion and documentation accuracy only; no behavior change.
- CLAUDE.md documented `pnpm build` as a three-stage pipeline ending in
pagefind. Pagefind does not exist in this repo (search is an external
Algolia crawler); the build is a single `next build --webpack`. Adds a
Vale install note, since `pnpm vale:check` is documented as required but
vale is a Go binary with no npm dependency.
- .gitignore reserved `public/toolkit-markdown/` for a build step that no
longer exists, and both .gitignore and the Makefile referenced
`make_toolkit_docs/`, a Python directory that was removed. `make
mcp-server-docs` was therefore a broken target.
- `data/toolkits/jira.json` was an unreferenced 133 KB copy at the repo
root; the live data is under `toolkit-docs-generator/data/toolkits/`.
- Drops unused dependencies (zustand, turndown, @mdx-js/react) and
redundant direct declarations that are supplied transitively
(@theguild/remark-mermaid via nextra, baseline-browser-mapping via next,
unist-util-visit-parents, mdast-util-to-string). Moves chalk to
devDependencies and consolidates the two colour libraries onto it.
- Adds @types/hast so neutralize-emails.tsx can use unist-util-visit
instead of a hand-rolled tree walk.
- The nightly generator workflow ran `pnpm build` with a working-directory
that has no package.json, so pnpm resolved upward and executed the root
Next production build. The step that follows runs the CLI through tsx and
needs no build.
- Renames ignored-toolkits.txt/excluded-toolkits.txt to
skip-toolkits.txt/remove-toolkits.txt, which say what they do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The header comment still listed "Build the toolkit docs generator" as step 1
after that step was removed. Renumber the remaining three.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercelBot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
docsReadyReadyPreviewAug 4, 2026 6:12pm

Request Review

@teallarson

Copy link
Copy Markdown
ContributorAuthor

@cursor review

teallarsonand others added 3 commits August 3, 2026 15:34
Two isolated behavior fixes.
Static rendering: the root layout awaited headers() to read "x-pathname"
and derive a locale. Awaiting headers() in the root layout opts the entire
route tree out of static rendering, so every page — including all 117
toolkit pages, which are pure functions of committed JSON — was
server-rendered on demand. The derived locale was always "en": proxy.ts
redirects every non-English locale to /en and getPreferredLocale returns
"en" unconditionally. The site paid full dynamic rendering to compute a
constant.
This is not an i18n change. TranslationBanner and the dictionary plumbing
stay in place; restoring real i18n means an app/[lang]/ route segment,
which is the correct Next pattern regardless.
Sitemap: app/sitemap.ts skips any directory whose name contains "[", which
is right for directory walking but meant all 117 toolkit pages were absent
from sitemap.xml — the largest content section on the site. Merges in
listValidIntegrationLinks() from app/_lib/toolkit-static-params.ts, the
same enumeration the integrations index uses, and dedupes against the
authored partner pages the disk walk already finds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eliminates the duplication that lets these subsystems drift.
Redirects were a 909-line array inside next.config.ts, so three consumers
regex-parsed the config as text — one of them carrying a second "reversed"
regex that existed only because a human might write {destination, source}
instead of {source, destination}, a problem that only exists when you parse
text instead of importing data. The array now lives in a typed redirects.ts
and every consumer imports it. Two further consumers that also parsed the
config as text (scripts/update-internal-links.ts and
tests/integration-index-links.test.ts) would have silently found zero
redirects, so they move to the import too. --auto-fix now appends to the
data file, and its six scattered "Auto-added redirects" marker blocks
collapse to one append point. The resolved array is byte-identical: 156
entries, same order.
check-redirects-utils.ts had 425 lines of tests but was imported by nothing
except its own test file, while the code that actually runs — the
pre-commit hook's scripts/check-redirects.ts — kept private copies of the
same six helpers. The tests guarded a copy while the shipping
implementation was untested. The module moves to scripts/lib/ and the
shipping script now imports it.
Toolkit primitives (data dir, toKebabCase, normalizeToolkitId, the category
list, the *Api heuristic, docsLink→slug) existed in 2-7 copies, one pair
carrying a "must stay in sync" comment. They collapse into
toolkit-docs-generator/src/shared/, which both halves can import. All seven
data-dir consumers now honor TOOLKIT_DATA_DIR; previously only two did.
The Node-only path resolution lives in its own module because client
components reach the primitives through the integrations index, and a
node:* import anywhere in that graph fails the webpack browser build.
The consumer-side contract was a four-field duck-check that never verified
tools was an array, followed by an unchecked cast — while toToolkitSummary
immediately calls data.tools.map(). The generator's Zod schemas are now the
single shared contract, the 522-line hand-written mirror is z.infer, and
zod moves to dependencies because it enters the app's runtime path.
Corruption is now loud and absence stays quiet: three catch blocks treated
missing, unparseable, and schema-invalid identically, so a malformed file
from the nightly PR silently dropped a toolkit and 404'd. Unparseable or
invalid now throws with the file path and the Zod issues, failing the
build. An unrecognized category throws instead of being coerced to
"others", which had no route directory and would have made every toolkit
in a new category a clickable card pointing at a 404.
One cache()-wrapped loader replaces 11 full passes over the data directory
per build, and removes the scan-every-file miss path that a burst of
unknown IDs could otherwise trigger at request time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Base automatically changed from fix/static-rendering-and-sitemap to mainAugust 4, 2026 13:06
@teallarsonteallarson changed the title refactor: make the toolkit data contract single and enforcedrefactor: singular + enforced toolkit data contractAug 4, 2026
@teallarson
teallarson marked this pull request as ready for review August 4, 2026 16:04
Comment threadapp/en/resources/integrations/_lib/toolkit-docs-page.tsx
Deduplicate production readToolkitData calls so generateMetadata and Page
share one in-flight read, recover failed cache entries, and drop stale
"others" references from broken-link-check and sidebar sync docs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Comment threadtoolkit-docs-generator/src/shared/toolkit-data-dir.ts
Comment thread.github/workflows/generate-toolkit-docs.yml Outdated
teallarsonand others added 2 commits August 4, 2026 13:53
Deriving DEFAULT_TOOLKIT_DATA_DIR from import.meta.url broke
/api/toolkit-data/[toolkitId] on deploy. Webpack replaces
import.meta.url with a compile-time constant — the source file's
absolute path on the build machine — so the resolved directory is
/vercel/path0/... and does not exist inside the deployed function.
Build-time static generation still succeeded, which is why nothing
failed locally or in CI; the preview returned 500 for every toolkit
data request while production returned 200.
Anchoring on process.cwd() restores the path production already uses,
and adds a test that chdirs before importing the module so a
self-relative default fails loudly instead of only on deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The jq program is single-quoted, so the shell passed the backslashes
through untouched and jq read `\\n` as an escaped backslash plus "n".
Slack rendered the whole alert as one line with literal `\n` in it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
app/sitemap.ts hand-rolled the same TOOLKIT_DATA_DIR expression the
shared resolver already provides, so the two could drift. The README
pointed at .github/scripts/sync-toolkit-sidebar.ts, which lives in
toolkit-docs-generator/scripts/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit bfdd661. Configure here.

Comment threadapp/_lib/toolkit-data.ts
readToolkitDataUncached resolves a lookup from two forms of its input:
the normalized id (direct file, byNormalizedId) and the lowercased raw
string (bySlug). Keying the cache on only the normalized form collapsed
variants that resolve differently, and NotionToolkit is a real instance
in the generated data — its file is notiontoolkit.json and "notion"
reaches it only through the slug map.
So on a warm instance the first variant requested decided the answer for
all of them. /api/toolkit-data/no-tion served the full Notion payload
where production 404s, and in the other order a miss pinned null so the
real Notion page lost its tool detail and markdown export. The route
takes arbitrary ids, so any request could trigger either direction.
Key on the lowercased id instead, and retain only lookups that found
data: concurrent callers still share the in-flight promise, but a
resolved miss is dropped so an absent toolkit resolves once generated
and arbitrary ids can't grow the map without bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@sdserranogsdserranog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@teallarson
teallarson merged commit ab9a282 into mainAug 5, 2026
9 checks passed
teallarson added a commit that referenced this pull request Aug 5, 2026
main squash-merged #1105 as ab9a282, and this branch predated the last
five commits of that PR, so the merge was a real three-way conflict
across 12 files rather than squash noise.
Resolution rule: main wins on the data-contract and runtime fixes it
added; S2 wins on the check enablement it exists to deliver.
- main wins wholesale on the 5 files S2 never touched: the toolkit data
cache keying, the cwd-anchored data-dir resolver, the sitemap resolver
share, the Slack alert newlines, and the page-factory comment.
- S2 wins on the .js import specifiers in scripts/. The generator project
is module: NodeNext with declaration + outDir, so it emits and .ts
specifiers are illegal there. main never caught this because its
tsconfig only included src/**.
- S2 wins on docsLink ?? null: ToolkitSlugSource declares
docsLink?: string | null and the project sets exactOptionalPropertyTypes,
so passing undefined is an error. Behaviour is unchanged.
- sync-toolkit-sidebar.test.ts took both sides: S2's required navGroup
field plus main's corrected categories, since the removed "others"
catch-all no longer exists.
Root and generator typecheck clean, 801 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@teallarson@sdserranog