Skip to content

perf(supervisor): sequential inspect loop in DockerResourceMonitor blocks task scheduling thread #3727

Description

@KanhaiyaBagul

Bug Description

The DockerResourceMonitor.getNodeResources() method calculates system resource consumption by iterating through all containers. During each iteration, it performs a sequential inspect call to retrieve CPU and memory limits:

for(constcontainerofstats){if(container.State==="running"){constc=this.docker.getContainer(container.Id);const{ HostConfig }=awaitc.inspect();// Sequential blocking callconstcpu=this.resourceParser.cpu(HostConfig.NanoCpus??0);constmemory=this.resourceParser.memory(HostConfig.Memory??0);cpuUsed+=cpu;memoryUsed+=memory;}}

Because getNodeResources is called frequently to make container placement decisions during scheduling, this sequential inspect loop acts as a synchronous bottleneck. On docker hosts running many containers, performing sequential inspect round-trips to the Unix socket blocks the scheduling thread for seconds and creates workload placement latency.

Root Cause File

apps/supervisor/src/resourceMonitor.ts

Steps to Reproduce

  1. Start the Trigger.dev supervisor daemon on a machine hosting 50+ active docker containers.
  2. Trigger multiple rapid task scheduling executions (wouldFit).
  3. Notice scheduling latency and elevated CPU thread blocking inside the supervisor daemon due to sequential docker socket round-trips.

Proposed Fix

Use Promise.all to fetch container configurations concurrently rather than sequentially:

construnningContainers=stats.filter(container=>container.State==="running");constinspections=awaitPromise.all(runningContainers.map(async(container)=>{constc=this.docker.getContainer(container.Id);const{ HostConfig }=awaitc.inspect();return{cpu: this.resourceParser.cpu(HostConfig.NanoCpus??0),memory: this.resourceParser.memory(HostConfig.Memory??0),};}));for(constinspectofinspections){cpuUsed+=inspect.cpu;memoryUsed+=inspect.memory;}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions