Queue callback fails with MultipartParseError on bodies >8KB delivered in multiple chunks (mixpart default maxBoundaryBuffer)

[robertnowell]

Summary

Queue callbacks delivered to POST /.well-known/workflow/v1/flow 5xx with MultipartParseError: Buffer size limit exceeded: <N> bytes > 8192 bytes whenever (a) the multipart body totals >8KB and (b) it arrives in more than one network chunk — which is the common case for any workflow with non-trivial state.

@vercel/queue@0.1.x calls mixpart's parseMultipartStream(response) with no options, so its maxBoundaryBuffer default of 8192 bytes applies. That default trips for almost any HTTP-delivered queue callback above ~8KB.

Cross-references: #1735 reports a different MultipartParseError mode (missing initial boundary) plus CBOR truncation in the same queue callback path. This issue is a third, distinct mode in the same surface.

Error Message (verbatim from production)

Queue callback error: Error [MultipartParseError]: Buffer size limit exceeded: 13319 bytes > 8192 bytes. This may indicate malformed multipart data with invalid boundaries.
 at r9.write (.next/server/chunks/0oz0__pnpm_0.dvwv8._.js:125:1522)
 at r9.startProducer (.next/server/chunks/0oz0__pnpm_0.dvwv8._.js:125:1034)

Status code: 500. Path: /.well-known/workflow/v1/flow. User-Agent: vercel-queue/v2beta.

In our environment 22 occurrences in 24h, tightly clustered at two chunk sizes (12,295 and 13,319 bytes — corresponding to two distinct workflow types).

Root Cause

In mixpart/dist/index.mjs (private workspace package, version 0.0.5):

// constructor of StreamingMultipartParser
this.maxBoundaryBuffer = options.maxBoundaryBuffer ?? 8192;

In the write() loop (lines ~263–268):

if (this.buffer !== null) {
 const newSize = this.buffer.length + chunkLength;
 const maxAllowedSize = this.state === 2 /* Header */ ? this.maxHeaderSize : this.maxBoundaryBuffer;
 if (newSize > maxAllowedSize) {
 throw new MultipartParseError(
 `Buffer size limit exceeded: ${newSize} bytes > ${maxAllowedSize} bytes. ...`
 );
 }
 // prepend buffer to chunk and continue parsing
 ...
}

The check fires whenever the parser carried over anything from a prior write() call (even an empty Uint8Array left after the opening boundary) and the next inbound chunk + buffer exceeds 8192 bytes. Because the parser buffers the post-boundary tail in Start and AfterBoundary states (lines 320–326, 386), the buffer is non-null almost immediately after the opening boundary is consumed. Any subsequent chunk >8KB then trips the cap regardless of total body size.

@vercel/queue@0.1.1, @vercel/queue@0.1.4 (and by inspection 0.1.6) all call:

// @vercel/queue/dist/index.mjs:1525, 1601 (and equivalents in newer versions)
for await (const multipartMessage of parseMultipartStream(response)) { ... }

with no options object, so the 8KB default applies.

Reproduction (10-line standalone)

import { parseMultipartStream } from "mixpart";

function craftedStream(body, contentType, firstChunkBytes) {
 const enc = new TextEncoder().encode(body);
 let i = 0, first = false;
 const stream = new ReadableStream({
 pull(c) {
 if (i >= enc.length) return c.close();
 const end = first ? enc.length : Math.min(i + firstChunkBytes, enc.length);
 first = true;
 c.enqueue(enc.slice(i, end));
 i = end;
 }
 });
 return new Response(stream, { headers: { "Content-Type": contentType } });
}

const boundary = "----TestBoundary";
const body =
 `--${boundary}\r\nContent-Type: application/octet-stream\r\n\r\n${"X".repeat(13000)}\r\n--${boundary}--\r\n`;

// First chunk just contains the opening boundary; second chunk is 13KB.
// Reproduces "Buffer size limit exceeded: ~13037 bytes > 8192 bytes".
for await (const part of parseMultipartStream(craftedStream(body, `multipart/form-data; boundary=${boundary}`, 37))) {
 await new Response(part.payload).arrayBuffer();
}

Passes if parseMultipartStream(stream, { maxBoundaryBuffer: 1024 * 1024 }) is used instead.

Production Symptom

Vercel Queue retries failed callbacks every ~5 min. Eventually a retry with different network framing happens to keep all chunks under 8KB and succeeds. So workflows do eventually complete, but start latency balloons by 15+ minutes — long after the user-facing generationExpectedDurationMs (10 min in our case) has timed out and the UI has reported failure.

A real workflow run that took 15m 33s end-to-end:

• 09:06:36 flow callback → 500 (MultipartParseError)
• 09:11:42 retry → 500 (same)
• 09:16:47 retry → 500 (same)
• 09:21:54 retry → 200, workflow finally started
• 09:22:08 workflow completed

Suggested Fixes (in order of preference)

1. @vercel/queue — pass an explicit maxBoundaryBuffer to parseMultipartStream. A 1 MiB ceiling is plenty for queue callbacks and gives ~80× headroom over observed callback chunk sizes (12–13KB).

   // packages where @vercel/queue calls parseMultipartStream
   for await (const m of parseMultipartStream(response, { maxBoundaryBuffer: 1024 * 1024 })) { ... }

2. mixpart — at minimum, the buffer-after-write check at line 263 should be a no-op when this.buffer is empty (length === 0), so a benign 0-byte leftover from AfterBoundary doesn't gate the next inbound chunk. Even better: raise the default to a sane HTTP-body number (1 MiB).

3. Both are by Timer <timer@vercel.com> per package.json author fields, so doing (1) is sufficient — the queue is the public surface.

Verified Workaround for Downstream Consumers

A pnpm patch mixpart@0.0.5 raising the default from 8192 to 1048576 fixes the issue in our deployment:

- this.maxBoundaryBuffer = options.maxBoundaryBuffer ?? 8192;
+ this.maxBoundaryBuffer = options.maxBoundaryBuffer ?? 1048576;

End-to-end A/B in production with the same 3.1 MB Mirai-Clinical workflow input (runEmailBatchWorkflow):

• Unpatched deployment: 1 MultipartParseError 500 + 6 lucky-framing 200s in 5min
• Patched deployment: 8 callbacks, 0 errors

Environment

• workflow: 4.2.0-beta.67
• @vercel/queue: 0.1.1 (also confirmed in installed 0.1.4, and by inspection of 0.1.6 on npm)
• mixpart: 0.0.5 (private workspace dep)
• Next.js 16.2.4 with withWorkflow()
• Deployed on Vercel (lambda + queue)
• Production deployment ID has been intentionally redacted; happy to share via private channel

[NathanColosimo]

This should have been fixed in #1987, which upgraded to vqs 0.1.7, mixpart 0.0.6, which should have fixed this issue. Let us know if you're still getting it! @robertnowell