Uh oh!
There was an error while loading. Please reload this page.
fix: upgrade @sentry/* to ^10.70.0 to fix span retention memory leaks - #1572
Merged
Conversation
The web process leaks ~570 MiB/h of old-space heap. In-prod sampling heap profiles show the retained memory is request-scoped tracing machinery — spans, OTel contexts, and the request/response objects they reference — with no app code among the top allocators. Upstream shipped fixes matching that signature exactly, all after our pinned 10.40.0: - 10.56.0: WeakRef for Span<->Scope circular references, and the sent-span cache moved to an LRUMap capped at 10k entries (#21242, closes #18339 "memory leak with high tracesSampleRate") - 10.49.0: WeakRef for OTel context stored on scope (#20328) - 10.43.0: vercel-ai tool-call span map leak (#19328) — the VercelAI integration is active in this process The running build demonstrably predates the fix: the live exporter's _sentSpans map measured 29,333 entries, three times the post-fix cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
WalkthroughThe backend and web packages upgrade Sentry dependencies to ChangesSentry dependency updates
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
License Audit
Weak Copyleft Packages (informational)
Resolved Packages (26)
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The web process leaks ~570 MiB/h of old-space JS heap (728 → 3589 MiB over 5h yesterday), until GC stalls trip the 1s liveness probe and the kubelet kills the container — the ~5–9h restart cadence on
app.sourcebot.dev. The heap-limit raise in #1569 removed the old pinned-at-ceiling thrash but, as flagged there, did not stop the growth.Root-causing this with in-prod V8 sampling heap profiles (inspector on the live
next-server) showed the retained memory is request-scoped tracing machinery — Sentry/OTelstartSpanand contextcloneallocations, Prisma client span wrappers, Nextreq/resobjects — with no app code among the top allocators and every guessable buffer (exporter span buckets, incremental cache, breadcrumbs, listeners) measured and bounded. Entire request graphs survive response completion.Fix
Bump
@sentry/nextjs,@sentry/node, and@sentry/profiling-nodefrom^10.40.0to^10.70.0. Upstream shipped fixes matching the measured signature exactly, all after 10.40.0:WeakReffor Span↔Scope circular references + sent-span cache converted to an LRUMap capped at 10k (closes#18339, "memory leak with high tracesSampleRate")WeakReffor OTel context stored on scope (contexts were our second-biggest retained cohort)VercelAIintegration is active in this processDirect evidence the running build predates the fix: probing the live exporter found
_sentSpansat 29,333 entries — three times the 10k cap #21242 introduced.tracesSampleRateis 0.1, but homepage traces run ~900 spans, so absolute span volume reaches the same failure mode as the "high tracesSampleRate" issue.Verification
@sentry/*packages resolve to 10.70.0; zero residual 10.40.x inyarn.lockyarn workspace @sourcebot/backend build— exit 0yarn workspace @sourcebot/web test run— 1239/1239 pass (106 files)yarn workspace @sourcebot/web build— compiles cleanIf the slope doesn't flatten
The fallback is the feature-gate bisect from the investigation: env-gate
profileSessionSampleRate(currently hardcoded 1.0 withprofileLifecycle: 'trace'), thenLocalVariablesAsync— both have their own upstream leak history and both are fed by the still-unfixed Sentry egressETIMEDOUTs (84–134/h), which remain worth fixing independently.🤖 Generated with Claude Code
Note
Cursor Bugbot is generating a summary for commit a38db5e. Configure here.
Summary by CodeRabbit