You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Workflow runs that fail inside the SDK's AES-GCM encryption layer were being misclassified as USER_ERROR. SDK-level decryption is never user code — the user never directly invokes subtle.decrypt — so failures here should be RUNTIME_ERROR.
This PR adds a new error class and wires it into the run-failure classifier, without addressing the root cause of the decryption failure itself (that investigation is ongoing). The change is intentionally narrow: better classification + diagnostic context, so the next mystery report is properly categorized and immediately actionable.
What I observed
Production reports surface as:
[Workflow] Error while running workflow {
workflowRunId: 'wrun_01KSG6WQKPKNMH37C401KYHQ9M',
errorCode: 'USER_ERROR',
errorName: 'OperationError',
errorStack: 'OperationError: The operation failed for an operation-specific reason\n' +
' at AESCipherJob.onDone (node:internal/crypto/util:646:19)'
}
OperationError from AESCipherJob.onDone is what Node's Web Crypto API throws when an AES-GCM auth-tag verification fails. The bare native DOMException doesn't match any of classifyRunError's RUNTIME_ERROR_CHECKS (which are name-based duck checks), so it falls through to USER_ERROR.
Changes
@workflow/errors (packages/errors/src/index.ts):
New RUNTIME_DECRYPTION_FAILED slug.
New RuntimeDecryptionError (extends WorkflowRuntimeError) with optional structured context (operation, byteLength, formatPrefix).
@workflow/core:
packages/core/src/encryption.ts: wrap both encrypt() and decrypt() Web Crypto calls; rewrap any failure as RuntimeDecryptionError with diagnostic context (printable or hex prefix of the input header, byte length, operation). The existing length-precheck now also throws RuntimeDecryptionError.
packages/core/src/serialization/encryption.ts & packages/core/src/serialization.ts: the two "encrypted-but-no-key" throw paths now use RuntimeDecryptionError.
packages/core/src/classify-error.ts: RuntimeDecryptionError.is added to RUNTIME_ERROR_CHECKS so classifyRunError routes these failures to RUNTIME_ERROR.
packages/core/src/classify-error.test.ts (extended): RuntimeDecryptionError → RUNTIME_ERROR, plus a documentation test that a bare native OperationError still classifies as USER_ERROR (proves the encryption module's wrap is what does the work).
All existing tests still pass:
@workflow/errors: 36/36 ✅
@workflow/core: 1024/1024 ✅
pnpm typecheck (full repo): 40/40 packages ✅
What this does NOT fix
The actual decryption failure. Root cause is still under investigation — see the prior analysis. Strongest current hypothesis remains transport-level corruption/truncation of ciphertext between storage and read (in particular the workflow-server /refs endpoint, where a guard against truncated bodies was prototyped on a feature branch but never landed on main).
The diagnostic context added here is specifically what we need to triangulate the source on the next occurrence: byte length distinguishes "truncated" from "tampered", and format prefix distinguishes "valid encr envelope with bad ciphertext" from "garbage bytes that happened to land in a decrypt path".
SDK-level AES-GCM encrypt/decrypt failures are never the user's fault,
but the run-failure classifier was tagging them as USER_ERROR because
the native Web Crypto OperationError (most commonly raised by
AESCipherJob.onDone on GCM auth-tag mismatch) does not match any
RUNTIME_ERROR_CHECKS entry.
Introduce a new RuntimeDecryptionError (subclass of WorkflowRuntimeError)
that the encryption module throws when subtle.encrypt/subtle.decrypt
fails, with the original DOMException as cause plus diagnostic context
(operation, byteLength, printable/hex format prefix of the input
header). classifyRunError now picks it up via RUNTIME_ERROR_CHECKS, so
these failures surface as RUNTIME_ERROR with a proper named class for
dashboards and triage.
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Workflow run failures originating in the SDK's AES-GCM encryption layer (most notably Node's native OperationError from AESCipherJob.onDone on GCM auth-tag mismatch) were falling through to USER_ERROR because classifyRunError's name-based duck checks didn't match a raw DOMException. This PR introduces a RuntimeDecryptionError (subclass of WorkflowRuntimeError) that the encryption module always wraps Web Crypto failures in, plus diagnostic context (operation, byte length, header prefix), so failures classify as RUNTIME_ERROR and carry enough telemetry to triangulate root cause on the next occurrence. No root-cause fix is attempted.
Changes:
New RuntimeDecryptionError class + runtime-decryption-failed slug in @workflow/errors with optional structured context.
Wrap encrypt/decrypt Web Crypto calls in packages/core/src/encryption.ts and rewrap the two "encrypted-but-no-key" throws in serialization paths.
Add RuntimeDecryptionError.is to RUNTIME_ERROR_CHECKS and cover the new behavior with errors + core tests.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
File
Description
packages/errors/src/index.ts
Adds RUNTIME_DECRYPTION_FAILED slug and RuntimeDecryptionError class with name-based .is().
Follow-up thought after tracing the runtime path: I think this PR is appropriately scoped to improving attribution (RUNTIME_ERROR rather than USER_ERROR), and it should not be required to solve retry behavior as part of this change.
That said, we should follow up by applying the same bounded-redelivery precedent used for replay timeouts to RuntimeDecryptionErrors encountered while replaying remotely fetched persisted data. An AES-GCM authentication failure is terminal for the bytes/key in the current attempt, so we must not continue execution; but if the bytes came from a transiently truncated or corrupted /refs response, a fresh queue delivery can re-fetch them successfully. Today we commit run_failed immediately, which turns a potentially recoverable read failure into a terminal workflow failure.
Concretely, for managed worlds we should let the queue redrive a small bounded number of times (re-fetching the events/ref payload each delivery), then commit terminal run_failed as RUNTIME_ERROR if the decryption failure persists. Longer term, detecting response truncation or integrity failure at the /refs transport boundary would let us classify the retryable case more directly. This feels like a focused follow-up PR rather than a blocker for the attribution fix here.
…x, propagate through serialization wrappers
Addresses review feedback on #2145:
- Add a RuntimeDecryptionError reducer/reviver (+ SerializableSpecial
entry + globalThis registration) so its `context` (operation,
byteLength, formatPrefix) survives the dehydrate/hydrate run-error
round trip instead of being dropped by the generic Error reducer.
- Stop capturing `formatPrefix` in the low-level encryption layer, which
only sees the stripped AES payload (nonce bytes), not the outer `encr`
marker. The serialization layer now attaches the real envelope prefix.
- Rethrow RuntimeDecryptionError unchanged from the serialize/dehydrate
catch blocks instead of reframing it as a SerializationError, so an
encryption failure during dehydration stays a RUNTIME_ERROR rather than
being misclassified as USER_ERROR.
Agreed on both points — keeping this PR scoped to attribution, and treating bounded redelivery as a focused follow-up.
The reasoning is sound: an AES-GCM auth failure is terminal for the current bytes/key, but if those bytes came from a transiently truncated/corrupted /refs response, a fresh queue delivery can re-fetch and succeed. Committing run_failed immediately turns a recoverable read failure into a terminal one.
I'll open a follow-up to apply the bounded-redelivery precedent (the same one used for replay timeouts) to RuntimeDecryptionErrors encountered while replaying remotely-fetched persisted data on managed worlds: redrive a small bounded number of times (re-fetching the events/ref payload each delivery), then commit terminal run_failed as RUNTIME_ERROR if it persists. The RuntimeDecryptionError class + diagnostic context landed here give that follow-up a clean signal to branch on, and the longer-term /refs transport-boundary integrity check would let us classify the retryable case even more directly.
The review feedback on this PR has been addressed in the latest commits:
RuntimeDecryptionError.context now round-trips through dehydrateRunError/hydrateRunError (reducer/reviver + globalThis registration).
formatPrefix is captured at the serialization layer (real encr marker) instead of the low-level layer (which only saw nonce bytes).
Encrypt-side failures now propagate as RuntimeDecryptionError through the dehydrate wrappers instead of being reframed as SerializationError (→ USER_ERROR).
- Mirror the catch/enrich/rethrow block from serialization/encryption.ts
around the stream-path aesGcmDecrypt() call so auth-tag failures on
encrypted stream frames also carry context.formatPrefix = 'encr'
(addresses review feedback). Add a tampered-frame test.
- Fix all auto-fixable Biome lint findings in the touched files
(template literals, useless try/catch wrappers, optional chaining,
non-null assertions).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Workflow runs that fail inside the SDK's AES-GCM encryption layer were being misclassified as
USER_ERROR. SDK-level decryption is never user code — the user never directly invokessubtle.decrypt— so failures here should beRUNTIME_ERROR.This PR adds a new error class and wires it into the run-failure classifier, without addressing the root cause of the decryption failure itself (that investigation is ongoing). The change is intentionally narrow: better classification + diagnostic context, so the next mystery report is properly categorized and immediately actionable.
What I observed
Production reports surface as:
OperationErrorfromAESCipherJob.onDoneis what Node's Web Crypto API throws when an AES-GCM auth-tag verification fails. The bare native DOMException doesn't match any ofclassifyRunError'sRUNTIME_ERROR_CHECKS(which are name-based duck checks), so it falls through toUSER_ERROR.Changes
@workflow/errors(packages/errors/src/index.ts):RUNTIME_DECRYPTION_FAILEDslug.RuntimeDecryptionError(extendsWorkflowRuntimeError) with optional structuredcontext(operation, byteLength, formatPrefix).@workflow/core:packages/core/src/encryption.ts: wrap bothencrypt()anddecrypt()Web Crypto calls; rewrap any failure asRuntimeDecryptionErrorwith diagnostic context (printable or hex prefix of the input header, byte length, operation). The existing length-precheck now also throwsRuntimeDecryptionError.packages/core/src/serialization/encryption.ts&packages/core/src/serialization.ts: the two "encrypted-but-no-key" throw paths now useRuntimeDecryptionError.packages/core/src/classify-error.ts:RuntimeDecryptionError.isadded toRUNTIME_ERROR_CHECKSsoclassifyRunErrorroutes these failures toRUNTIME_ERROR.Test coverage
packages/errors/src/runtime-decryption-error.test.ts(new, 6 tests): name, inheritance, docs URL, cause preservation, context shape, name-basedis()duck check.packages/core/src/encryption.test.ts(new, 8 tests): happy-path round-trip, length-check failure, GCM auth-tag tamper →RuntimeDecryptionError(cause = OperationError), wrong-key decryption → same, encrypt-only key used for encrypt →RuntimeDecryptionError, printable + hex format-prefix capture.packages/core/src/classify-error.test.ts(extended):RuntimeDecryptionError → RUNTIME_ERROR, plus a documentation test that a bare nativeOperationErrorstill classifies asUSER_ERROR(proves the encryption module's wrap is what does the work).All existing tests still pass:
@workflow/errors: 36/36 ✅@workflow/core: 1024/1024 ✅pnpm typecheck(full repo): 40/40 packages ✅What this does NOT fix
The actual decryption failure. Root cause is still under investigation — see the prior analysis. Strongest current hypothesis remains transport-level corruption/truncation of ciphertext between storage and read (in particular the workflow-server
/refsendpoint, where a guard against truncated bodies was prototyped on a feature branch but never landed onmain).The diagnostic context added here is specifically what we need to triangulate the source on the next occurrence: byte length distinguishes "truncated" from "tampered", and format prefix distinguishes "valid
encrenvelope with bad ciphertext" from "garbage bytes that happened to land in a decrypt path".