The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

Description

@os-trump

Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

The mechanism, measured on #14832

Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

Measured directly, from inside a child whose stderr was a pipe nobody read:

1. before touching process.stderr: O_NONBLOCK=false
2. after materialising process.stderr: O_NONBLOCK=true
3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
4. after handle.setBlocking(false): O_NONBLOCK=true ← restored

and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

#14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

Where the published path does it

bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

filelinestdio
packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

Why this matters beyond a test

src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

What would settle it, cheaply

⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

  1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
  2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

Not to be confused with

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
       blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
      }
      } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
      })();
      (function(){
      try {
      var __m = "github.com";
      var __re = new RegExp('^' + "github\\.com" + '
      
      Skip to content

      The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

      Description

      @os-trump

      Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

      The mechanism, measured on #14832

      Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

      Measured directly, from inside a child whose stderr was a pipe nobody read:

      1. before touching process.stderr: O_NONBLOCK=false
      2. after materialising process.stderr: O_NONBLOCK=true
      3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
      4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
      

      and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

      #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

      Where the published path does it

      bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

      filelinestdio
      packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
      packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
      packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
      packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
      packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
      packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

      os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

      Why this matters beyond a test

      src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

      the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

      That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

      What would settle it, cheaply

      ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

      1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
      2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

      If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

      Not to be confused with

      Activity

      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

      Metadata

      Metadata

      Assignees

      No one assigned

        Labels

        No labels
        No labels

        Type

        No type

        Projects

        No projects

          Milestone

          No milestone

          Relationships

          None yet

          Development

          No branches or pull requests

          Issue actions

          , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
          Skip to content

          The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

          Description

          @os-trump

          Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

          The mechanism, measured on #14832

          Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

          Measured directly, from inside a child whose stderr was a pipe nobody read:

          1. before touching process.stderr: O_NONBLOCK=false
          2. after materialising process.stderr: O_NONBLOCK=true
          3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
          4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
          

          and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

          #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

          Where the published path does it

          bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

          filelinestdio
          packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
          packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
          packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
          packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
          packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
          packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

          os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

          Why this matters beyond a test

          src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

          the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

          That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

          What would settle it, cheaply

          ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

          1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
          2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

          If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

          Not to be confused with

          Activity

          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

          Metadata

          Metadata

          Assignees

          No one assigned

            Labels

            No labels
            No labels

            Type

            No type

            Projects

            No projects

              Milestone

              No milestone

              Relationships

              None yet

              Development

              No branches or pull requests

              Issue actions

              , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
              Skip to content

              The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

              Description

              @os-trump

              Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

              The mechanism, measured on #14832

              Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

              Measured directly, from inside a child whose stderr was a pipe nobody read:

              1. before touching process.stderr: O_NONBLOCK=false
              2. after materialising process.stderr: O_NONBLOCK=true
              3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
              4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
              

              and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

              #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

              Where the published path does it

              bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

              filelinestdio
              packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
              packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
              packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
              packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
              packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
              packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

              os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

              Why this matters beyond a test

              src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

              the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

              That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

              What would settle it, cheaply

              ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

              1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
              2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

              If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

              Not to be confused with

              Activity

              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

              Metadata

              Metadata

              Assignees

              No one assigned

                Labels

                No labels
                No labels

                Type

                No type

                Projects

                No projects

                  Milestone

                  No milestone

                  Relationships

                  None yet

                  Development

                  No branches or pull requests

                  Issue actions

                  , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
                  Skip to content

                  The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

                  Description

                  @os-trump

                  Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

                  The mechanism, measured on #14832

                  Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

                  Measured directly, from inside a child whose stderr was a pipe nobody read:

                  1. before touching process.stderr: O_NONBLOCK=false
                  2. after materialising process.stderr: O_NONBLOCK=true
                  3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
                  4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
                  

                  and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

                  #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

                  Where the published path does it

                  bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

                  filelinestdio
                  packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
                  packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
                  packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
                  packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
                  packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
                  packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

                  os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

                  Why this matters beyond a test

                  src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

                  the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

                  That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

                  What would settle it, cheaply

                  ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

                  1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
                  2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

                  If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

                  Not to be confused with

                  Activity

                  Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                  Metadata

                  Metadata

                  Assignees

                  No one assigned

                    Labels

                    No labels
                    No labels

                    Type

                    No type

                    Projects

                    No projects

                      Milestone

                      No milestone

                      Relationships

                      None yet

                      Development

                      No branches or pull requests

                      Issue actions

                      , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
                      Skip to content

                      The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

                      Description

                      @os-trump

                      Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

                      The mechanism, measured on #14832

                      Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

                      Measured directly, from inside a child whose stderr was a pipe nobody read:

                      1. before touching process.stderr: O_NONBLOCK=false
                      2. after materialising process.stderr: O_NONBLOCK=true
                      3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
                      4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
                      

                      and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

                      #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

                      Where the published path does it

                      bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

                      filelinestdio
                      packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
                      packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
                      packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
                      packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
                      packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
                      packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

                      os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

                      Why this matters beyond a test

                      src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

                      the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

                      That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

                      What would settle it, cheaply

                      ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

                      1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
                      2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

                      If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

                      Not to be confused with

                      Activity

                      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                      Metadata

                      Metadata

                      Assignees

                      No one assigned

                        Labels

                        No labels
                        No labels

                        Type

                        No type

                        Projects

                        No projects

                          Milestone

                          No milestone

                          Relationships

                          None yet

                          Development

                          No branches or pull requests

                          Issue actions

                          , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
                          Skip to content

                          The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

                          Description

                          @os-trump

                          Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

                          The mechanism, measured on #14832

                          Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

                          Measured directly, from inside a child whose stderr was a pipe nobody read:

                          1. before touching process.stderr: O_NONBLOCK=false
                          2. after materialising process.stderr: O_NONBLOCK=true
                          3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
                          4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
                          

                          and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

                          #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

                          Where the published path does it

                          bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

                          filelinestdio
                          packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
                          packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
                          packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
                          packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
                          packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
                          packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

                          os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

                          Why this matters beyond a test

                          src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

                          the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

                          That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

                          What would settle it, cheaply

                          ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

                          1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
                          2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

                          If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

                          Not to be confused with

                          Activity

                          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                          Metadata

                          Metadata

                          Assignees

                          No one assigned

                            Labels

                            No labels
                            No labels

                            Type

                            No type

                            Projects

                            No projects

                              Milestone

                              No milestone

                              Relationships

                              None yet

                              Development

                              No branches or pull requests

                              Issue actions

                              , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
                              Skip to content

                              The PUBLISHED CLI puts its own stdout/stderr on the blocking write path every time it spawns a child with inherited stdio #14874

                              Description

                              @os-trump

                              Filed by the domain:cli dev on #14832 (session session_016yfqQh2dBgPAymYd7xipza), out of scope for that card and deliberately not fixed there. Unassigned and ungraded — triage's.

                              The mechanism, measured on #14832

                              Node sets O_NONBLOCK on fd 2 when it opens the pipe, so a write to a reader that has stopped reading is BUFFERED rather than parking the thread. libuv clears that flag again in the pre-exec of any child spawned with inherited stdio — deliberately, because a child expects blocking stdio — and inheriting is dup2, so the flag lives on an open file description the spawner shares. Clearing it for the child clears it for the spawner too.

                              Measured directly, from inside a child whose stderr was a pipe nobody read:

                              1. before touching process.stderr: O_NONBLOCK=false
                              2. after materialising process.stderr: O_NONBLOCK=true
                              3. after spawnSync(node -e 0, { stdio: 'inherit' }): O_NONBLOCK=false ← the clearing
                              4. after handle.setBlocking(false): O_NONBLOCK=true ← restored
                              

                              and the consequence, in a matched A/B on the same fixture (only step 4 differs): left blocking, the writer parks in write(2) on the first full-pipe write, 0 interval ticks fire, the event loop is dead and only a SIGKILL ends it; restored, 29 ticks fire, 192 KiB sit pending in userland, the process exits on its own.

                              #14832 is that hang reaching bin/run-dev.js through tsx, which spawns the esbuild service on a cold transform cache. The published CLI does its own spawning, so it does not need tsx to get there.

                              Where the published path does it

                              bin/run.js ships (it is the bin target; files names only dist, README.md, CHANGELOG.md), and the commands behind it spawn with inherited stdio in at least:

                              filelinestdio
                              packages/cli/src/commands/dev.ts221{ stdio: 'inherit' }
                              packages/cli/src/commands/dev.ts470{ stdio: ['inherit', 'inherit', 'inherit', 'ipc'] }
                              packages/cli/src/commands/dev.ts582execSync(command, { stdio: 'inherit', cwd })
                              packages/cli/src/commands/start.ts241, 444{ stdio: 'inherit', … }
                              packages/cli/src/commands/environments/bind.ts84{ stdio: 'inherit', env: process.env }
                              packages/cli/src/commands/init.ts873execSync(pm install, { stdio: 'inherit', … })

                              os dev spawning os serve is the headline case: after that spawn, the long-lived parent's OWN stdout and stderr are on the blocking path for the rest of the run.

                              Why this matters beyond a test

                              src/utils/format.ts already states the hazard in its own words, as a reason to refuse setBlocking(true):

                              the same binary runs os serve / os dev, and a blocking write to a pipe whose reader is slow blocks the event loop — it would trade truncated JSON for a server that stalls on its own logs.

                              That refusal is correct and is not the issue. The issue is that the premise it protects — that nobody has put stderr in blocking mode — is not actually held on the published path: the CLI turns it on itself, as a side effect of spawning, with nothing in any file saying so. Any consumer that pipes os dev and reads it slowly (a CI log collector, a backgrounded runner, a supervisor that stops draining while it does work) can park the CLI in the kernel. It is not a crash and not a timeout: the process is alive, idle, and unresponsive, with an empty log.

                              What would settle it, cheaply

                              ⛔ Not prescribing the fix. Two measurements decide whether this is real in the field rather than only in principle:

                              1. Run os dev from the built CLI with its stdout/stderr piped to a reader that stops reading after the spawn, and sample /proc/PID/syscall plus /proc/PID/fdinfo/1 and .../2. The reading to look for is syscall=1(write) on the main thread with O_NONBLOCK=false.
                              2. Do the same with the spawn changed to pipe + manual forwarding, as the negative control.

                              If it reproduces, the repair is the same shape #14832 landed — re-assert non-blocking mode on the write path — but on the published entry point, where it needs its own review, its own pin and a changeset that actually ships. That is precisely why it is a separate card and not a rider: #14832 is a p1 queue blocker and its diff should not also carry a shipped behaviour change to os dev.

                              Not to be confused with

                              Activity

                              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                              Metadata

                              Metadata

                              Assignees

                              No one assigned

                                Labels

                                No labels
                                No labels

                                Type

                                No type

                                Projects

                                No projects

                                  Milestone

                                  No milestone

                                  Relationships

                                  None yet

                                  Development

                                  No branches or pull requests

                                  Issue actions