Uh oh!
There was an error while loading. Please reload this page.
fix(algolia): add concurrency limit to indexing script - #17722
Merged
Conversation
…ript The script was firing Promise.all on all ~9,820 pages simultaneously, loading every HTML file into memory at once. This caused OOM kills on GitHub Actions runners (7 GB RAM). Add p-limit(50) to cap concurrent page processing, and add timing metrics so we can monitor performance in CI logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Uh oh!
There was an error while loading. Please reload this page.
- Cache algolia records per page using MD5 hash of HTML content. Cache lives in .next/cache/algolia-records/ (already in GH Actions cache). Subsequent runs only reprocess pages whose HTML changed. - Add Sentry metrics (pages, records, duration, cache hit/miss rate) via ALGOLIA_SENTRY_DSN env var (no-op if unset). - Use template literals instead of printf-style format specifiers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
- Add @sentry/node as direct dependency (was only transitive, TS couldn't find it) - Include frontmatter metadata in cache key so keyword-only changes invalidate the cache - Isolate cache write in its own try/catch so a write failure doesn't discard valid records Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
- Switch to @sentry/nextjs which re-exports @sentry/node, removing the need for a separate dependency - Use correct metrics API: attributes instead of tags - Add unit annotations to duration metrics - Add docs_type and commit_sha attributes to all metrics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fb84687. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Track which cache files are used during processing and delete any orphaned files afterward. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The add/remove of @sentry/node left transitive dep version drift in the lockfile. Restore the original. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
rahulchhabria
approved these changes
May 12, 2026
Uh oh!
There was an error while loading. Please reload this page.
sentrivana pushed a commit
that referenced
this pull request
May 18, 2026
## DESCRIBE YOUR PR Follow-up to #17711. The previous PR fixed `NODE_OPTIONS` causing `next build` to OOM, but the algolia script itself was also a problem — it fired `Promise.all` on all ~9,820 pages simultaneously with no concurrency limit, no caching, and no monitoring. **Changes:** 1. **Concurrency limit** — `p-limit(50)` caps concurrent page processing (same pattern as `generate-md-exports.mjs`). Previously all 9,820 pages were processed simultaneously via unbounded `Promise.all`. 2. **Content-hash caching** — MD5 hash of each HTML file is used as a cache key. Algolia records for unchanged pages are read from `.next/cache/algolia-records/` (already covered by the GitHub Actions cache step). First run processes all pages; subsequent runs only reprocess pages whose HTML changed. 3. **Sentry metrics** — tracks page count, record count, generation duration, and cache hit/miss rate via `ALGOLIA_SENTRY_DSN` env var. No-op if the secret isn't set. **Context:** - 212 failed runs burned **157 hours** of CI time (~$75) since late March - The `generate-md-exports.mjs` script processes the same HTML files but uses worker threads, `p-limit`, and caching — the algolia script had none of these - Developer docs (304 pages) always succeeded; user docs (9,820 pages → 234K records) consistently OOMed **Note:** The `ALGOLIA_SENTRY_DSN` secret needs to be added to the repo for metrics to flow. Without it, Sentry init is skipped and the metrics calls are no-ops. ## IS YOUR CHANGE URGENT? - [ ] Urgent deadline (GA date, etc.): - [x] Other deadline: Algolia search index hasn't updated since April 28 - [ ] None: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it. Thanks in advance for your help! ## PRE-MERGE CHECKLIST - [x] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIBE YOUR PR
Follow-up to #17711. The previous PR fixed
NODE_OPTIONScausingnext buildto OOM, but the algolia script itself was also a problem — it firedPromise.allon all ~9,820 pages simultaneously with no concurrency limit, no caching, and no monitoring.Changes:
Concurrency limit —
p-limit(50)caps concurrent page processing (same pattern asgenerate-md-exports.mjs). Previously all 9,820 pages were processed simultaneously via unboundedPromise.all.Content-hash caching — MD5 hash of each HTML file is used as a cache key. Algolia records for unchanged pages are read from
.next/cache/algolia-records/(already covered by the GitHub Actions cache step). First run processes all pages; subsequent runs only reprocess pages whose HTML changed.Sentry metrics — tracks page count, record count, generation duration, and cache hit/miss rate via
ALGOLIA_SENTRY_DSNenv var. No-op if the secret isn't set.Context:
generate-md-exports.mjsscript processes the same HTML files but uses worker threads,p-limit, and caching — the algolia script had none of theseNote: The
ALGOLIA_SENTRY_DSNsecret needs to be added to the repo for metrics to flow. Without it, Sentry init is skipped and the metrics calls are no-ops.IS YOUR CHANGE URGENT?
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST