Skip to content

fix(execution): cap isolate memory at 128MB and recycle workers every 200 executions#4543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
waleedlatif1 merged 4 commits into from
May 10, 2026
Merged

fix(execution): cap isolate memory at 128MB and recycle workers every 200 executions #4543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ec435c3
fix(execution): cap isolate memory at 128MB and recycle workers every…
waleedlatif1 May 10, 2026
66c9d59
fix(execution): set IVM_MAX_EXECUTIONS_PER_WORKER env default to 100
waleedlatif1 May 10, 2026
525f193
fix(execution): raise MAX_EXECUTIONS_PER_WORKER from 100 to 200
waleedlatif1 May 10, 2026
eb6068b
fix(execution): update memory limit error messages from 256 MB to 128 MB
waleedlatif1 May 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/sim/lib/core/config/env.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -248,7 +248,7 @@ export const env = createEnv({
IVM_DISTRIBUTED_MAX_INFLIGHT_PER_OWNER:z.string().optional().default('2200'), // Max owner in-flight leases across replicas
IVM_DISTRIBUTED_LEASE_MIN_TTL_MS: z.string().optional().default('120000'), // Min TTL for distributed in-flight leases (ms)
IVM_QUEUE_TIMEOUT_MS: z.string().optional().default('300000'), // Max queue wait before rejection (ms)
IVM_MAX_EXECUTIONS_PER_WORKER: z.string().optional().default('500'), // Max lifetime executions before worker is recycled
IVM_MAX_EXECUTIONS_PER_WORKER: z.string().optional().default('200'), // Max lifetime executions before worker is recycled
Comment thread
waleedlatif1 marked this conversation as resolved.
IVM_MAX_BROKER_ARGS_JSON_CHARS: z.string().optional().default('262144'), // Max JSON payload size for sandbox task broker args (isolate→host)
IVM_MAX_BROKER_RESULT_JSON_CHARS: z.string().optional().default('16777216'),// Max JSON payload size for sandbox task broker results (host→isolate)
IVM_MAX_BROKERS_PER_EXECUTION: z.string().optional().default('1000'), // Max broker calls per sandbox task execution
Expand Down
8 changes: 4 additions & 4 deletions apps/sim/lib/execution/isolated-vm-worker.cjs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -183,7 +183,7 @@ async function executeCode(request, executionId) {
const externalCopies = []

try {
isolate = new ivm.Isolate({ memoryLimit: 256 })
isolate = new ivm.Isolate({ memoryLimit: 128 })
if (executionId !== undefined) activeIsolates.set(executionId, isolate)
context = await isolate.createContext()
const jail = context.global
Expand DownExpand Up@@ -388,7 +388,7 @@ async function executeCode(request, executionId) {
stdout,
error: {
message:
'Execution exceeded memory limit (256 MB). Reduce image sizes or split the work into smaller batches.',
'Execution exceeded memory limit (128 MB). Reduce image sizes or split the work into smaller batches.',
name: 'MemoryLimitError',
},
}
Expand DownExpand Up@@ -529,7 +529,7 @@ async function executeTask(request, executionId) {
let tPhase = tStart

try {
isolate = new ivm.Isolate({ memoryLimit: 256 })
isolate = new ivm.Isolate({ memoryLimit: 128 })
Comment thread
waleedlatif1 marked this conversation as resolved.
if (executionId !== undefined) activeIsolates.set(executionId, isolate)
context = await isolate.createContext()
const jail = context.global
Expand DownExpand Up@@ -945,7 +945,7 @@ async function executeTask(request, executionId) {
stdout,
error: {
message:
'Execution exceeded memory limit (256 MB). Reduce image sizes or split the work into smaller batches.',
'Execution exceeded memory limit (128 MB). Reduce image sizes or split the work into smaller batches.',
name: 'MemoryLimitError',
},
timings,
Expand Down
2 changes: 1 addition & 1 deletion apps/sim/lib/execution/isolated-vm.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -128,7 +128,7 @@ const DISTRIBUTED_MAX_INFLIGHT_PER_OWNER =
Number.parseInt(env.IVM_DISTRIBUTED_MAX_INFLIGHT_PER_OWNER) ||
MAX_ACTIVE_PER_OWNER + MAX_QUEUED_PER_OWNER
const DISTRIBUTED_LEASE_MIN_TTL_MS = Number.parseInt(env.IVM_DISTRIBUTED_LEASE_MIN_TTL_MS) || 120000
const MAX_EXECUTIONS_PER_WORKER = Number.parseInt(env.IVM_MAX_EXECUTIONS_PER_WORKER) || 500
const MAX_EXECUTIONS_PER_WORKER = Number.parseInt(env.IVM_MAX_EXECUTIONS_PER_WORKER) || 200
Comment thread
waleedlatif1 marked this conversation as resolved.
const MAX_BROKER_ARGS_JSON_CHARS = Number.parseInt(env.IVM_MAX_BROKER_ARGS_JSON_CHARS) || 262_144
const MAX_BROKER_RESULT_JSON_CHARS =
Number.parseInt(env.IVM_MAX_BROKER_RESULT_JSON_CHARS) || 16_777_216
Expand Down
Loading