Reducing memory foot print for vercel AI messages

[JPeer264]

Problem Statement

When sendDefaultPii: true, requestMessagesFromPrompt parses and re-serializes ai.prompt.messages on every AI span, even when no transformation is needed.

For a 4mb prompt, this briefly doubles memory usage and has inside the function another memory footprint (which is unavoidable I guess):

1. Original JSON string on the span attribute
2. Parsed array from JSON.parse
3. Re-serialized string from getJsonString/getTruncatedJsonString

The parse + reserialize is only necessary when:

• System instructions need to be extracted (messages contain role: "system")
• Truncation is enabled

In the common case (no system messages, no truncation), we could skip parsing entirely and reuse the original string.

Solution Brainstorm

Not sure if there is a good solution for it or if it is worth fixing as 4mb is unusually big, but if there is a high traffic this could spike real fast.

Additional Context

No response

Priority

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it.}

[linear-code]

JS-2640

[HazAT]

👍
It can be especially tricky in constrained environments like CF or serverless where memory can be quite limited.
Holding multiple unnecessary copies of large strings in memory could be the difference between OOM and nothing.

[AyaanFaisal21]

Hey there! I thought this was an interesting thread and took the liberty to look into it myself. I opened a small PR for the truncation-off case: #21354.

The part that seemed worth targeting is the ai.prompt.messages string path when truncation is disabled. The SDK still needs to parse the JSON to extract system instructions and record the message count, but if extractSystemInstructions leaves the messages array unchanged, re-serializing the same large payload creates an avoidable full-size copy.

The PR reuses the original string only when nothing was transformed and truncation is off. The messier truncation-enabled and system-extraction paths are left untouched, so the scope stays fairly small.

On a synthetic ~4 MB no-system payload, I measured one full-size copy removed from this path with byte-equivalent output. I’d value your read on whether reusing the SDK’s original serialized string is acceptable for that attribute. I'm happy to adjust the scope or close it out if you’d rather fold this into a broader change.

[andreiborza]

Hi @AyaanFaisal21, thanks for taking the time to look into this and creating a PR. We'll have a look.