fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers

[matt-aitken]

Summary

Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's Int64..UInt64 range. Recovery stays purely reactive — the existing sanitizeRows walker just gains an extra branch, so the hot replication path pays nothing on healthy rows.

Fixes the still-firing customer-facing symptom from TRI-9755: a small number of runs continued to be stranded in EXECUTING on the Tasks page after #3708 deployed. CloudWatch showed Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix firing 8/8 times since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (117039831458782873093 → 117039831458782870000) — the precision-lossy value still serialises as a bare integer that exceeds UInt64.MAX, which ClickHouse rejects with INCORRECT_DATA.

How the bug ships

The customer task emits an output containing a Poshmark profile's spec_format:

{"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"}

That value is 1.17e20 — comfortably above UInt64.MAX (1.84e19) but comfortably below 1e21. Number.prototype.toString only switches to exponential form at |value| >= 1e21, so JSON.stringify emits the bare token 117039831458782870000 and the ClickHouse JSON(max_dynamic_paths) column fails with:

Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x)

Same error verbatim as prod. The same number quoted ("117039831458782870000") inserts fine — ClickHouse's dynamic JSON column accepts a String subtype on the same path.

What changed

apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts:

• New private isUnsafeJsonInteger(value) helper — true iff value is a finite integer-valued JS Number where |value| < 1e21 (so JSON.stringify emits integer form, not exponent) andvalue falls outside [Int64.MIN, UInt64.MAX].
• sanitizeUnknownInPlace gains a number-branch: when the predicate holds, replace the Number with String(value). The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway).
• Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — JSON.stringify emits them with exponents or as null, both of which ClickHouse accepts.

apps/webapp/test/sanitizeRowsOnParseError.test.ts: four new unit tests + an extension to sanitizeRows covering surrogate + integer fixes counted together across rows. The unit suite now covers:

• Positive value above UInt64.MAX (117039831458782870000 — the actual prod value)
• Negative value below Int64.MIN
• Boundary values pass through (42, Number.MAX_SAFE_INTEGER, 2^63)
• Non-integer numbers untouched (floats, 1e25, NaN, Infinity)
• The actual scan-social-profiles nested shape — finds the offending gp_id deep inside output.data.profiles[].spec_format[].platform_variables[].value

.server-changes/runs-replication-bigint-recovery.md — release notes entry.

Why reactive, not pre-flight

#prepareJson runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. sanitizeRows only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row.

Verification

• Reproduced 1:1 in a throwaway Docker clickhouse/clickhouse-server:25.12.11.4 (closest available to the prod 25.12.1.1579 build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with gp_id stored as a String subtype.
• pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts — 22/22 passing (18 existing + 4 new).
• pnpm run typecheck --filter webapp — clean.

Test plan

• pnpm run typecheck --filter webapp
• Unit tests pass against new + existing cases
• End-to-end Docker ClickHouse repro confirms recovery
• Post-deploy: confirm Sanitizing batch after ClickHouse JSON parse error warns fire instead of Dropped batch … errors when scan-social-profiles outputs trip CH again
• Post-deploy: confirm permanentlyDroppedBatches counter stops climbing in /stp/trigger-app-prod/ecs/replication/service-container/process-logs

What this does NOT do

• Doesn't backfill the ~120k+ existing stranded EXECUTING rows in production. Same as fix(webapp): recover from ClickHouse JSON parse failures in runs replication #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix[Docs] Add initial files and configuration #3).
• Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: b2cc562

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Hidden review stack artifact:

Walkthrough

This pull request extends the runs-replication row sanitizer to handle out-of-range integers that JSON-serialize as bare integer tokens. The change adds an isUnsafeJsonInteger predicate to detect JavaScript Numbers outside ClickHouse's Int64/UInt64 range, and updates the sanitizeUnknownInPlace walker to convert such values to their string form, preventing INCORRECT_DATA parser failures on dynamic JSON columns. The implementation includes comprehensive test coverage for safe/unsafe boundary conditions and nested object scanning, plus a changelog entry documenting the reactive recovery mechanism.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR objectives are entirely unrelated to the linked issue #3 about documentation setup (Mintlify configuration). The actual PR implements ClickHouse JSON parse error recovery with no connection to documentation.	Clarify the actual linked issue(s) this PR addresses. The linked issue #3 appears to be incorrect; verify the correct issue number for this ClickHouse recovery fix (likely TRI-9755).
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The PR description is comprehensive and well-structured, covering the bug, solution, testing, and trade-offs, but lacks the required checklist items and screenshots sections from the repository template.	Add the required checklist section with items marked complete/incomplete, verify the PR title follows convention, and include any relevant screenshots if applicable.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title accurately and concisely describes the main change: adding recovery from ClickHouse JSON parse failures caused by out-of-range integers.
Out of Scope Changes check	✅ Passed	All changes are tightly scoped to the ClickHouse JSON parse error recovery: modifications to the sanitizer, comprehensive test coverage, and a release notes entry directly support the stated objective.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/tri-9755-runs-replication-bigint-recovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}