Remote server update fails on Linux when node-pty must compile (no linux-x64 prebuild; opaque error)

[MayberryDT]

Summary

Remote/boot-service self-update can fail on Linux while preparing a pinned t3@<version> runtime because node-pty has no linux-x64 prebuild and the native rebuild fails. The desktop/UI only shows:

Server update failed: Could not prepare t3@0.0.33.

The underlying npm/node-gyp error is not surfaced, so operators have to dig through server npm debug logs and effect traces.

Environment

• Host: Ubuntu 24.04 (Linux x64)
• T3 server mode: boot-service / headless (t3 service, T3CODE_HOME pinned runtimes under runtime/versions/)
• From version: 0.0.32
• Target version: 0.0.33
• Node: v22.23.1
• System compiler: g++ → g++-13 (no g++-11 package installed)

What happens

1. Client requests remote update to exact version 0.0.33.
2. Server runs ensurePinnedRuntimeInstalled → npm install --prefix <staging> --no-fund --no-audit t3@0.0.33.
3. Install pulls node-pty@1.1.0.
4. node-pty install script: node scripts/prebuild.js || node-gyp rebuild.
5. Prebuild check: no prebuilds/linux-x64 in the published package (only darwin/win32 prebuilds present).
6. Rebuild fails. In our case the make step looked for g++-11 and exited 127.
7. npm exits 1 → PinnedRuntimeInstallError → UI: Could not prepare t3@0.0.33.

Relevant packaging observation on node-pty@1.1.0 tarball contents: prebuilds exist for darwin-arm64, darwin-x64, win32-arm64, win32-x64 only.

Expected

1. Linux installs should not require a local C++ toolchain for a normal server update (ship linux-x64 prebuilds for node-pty, or vendor a prebuilt binary for the supported Node ABI).
2. If native compile still fails, the remote update error should include the actionable cause (e.g. missing compiler / node-gyp stderr tail), not only Could not prepare t3@X.

Workaround that unblocked us

# stage with explicit compiler
CXX=g++ CC=gcc npm install --prefix ~/.t3/runtime/versions/.staging-... --no-fund --no-audit t3@0.0.33
# then t3 service update / normal activation

# prevent recurrence on the user service
# ~/.config/systemd/user/t3code.service.d/40-native-build.conf
# [Service]
# Environment=CXX=g++
# Environment=CC=gcc

Why this matters

Headless / remote Linux boxes are a first-class T3 deployment path (t3 service, Tailscale, mobile/desktop remote control). Forcing an opaque native compile on every server version bump makes remote updates fragile and hard to support.

Nice-to-haves

• Document remote-update prerequisites (Node version, compiler, disk) if native build remains required on some platforms.
• Consider caching successful node-pty builds across staged versions on the same Node ABI to avoid rebuild thrash.

Happy to provide redacted npm debug log excerpts if useful.

[keithce]

Confirmed another instance of this failure through t3 triage, with a different native-build trigger and an exact actionable cause.

Environment

• Server: t3code.service systemd user background service
• Server version: t3@0.0.37-nightly.20260829.1219
• Target: t3@0.0.38-nightly.20260831.1236
• Server host: Linux x64, Node 24.20.0
• Client: macOS nightly desktop app 0.0.38-nightly.20260831.1236
• Connection: remote through T3 Connect

Confirmed failure

Three remote-update attempts failed identically. The UI only reported:

Server update failed: Could not prepare t3@0.0.38-nightly.20260831.1236.

The server trace recorded PinnedRuntimeInstallError with npm exit code 1. Matching npm debug logs showed node-pty@1.1.0 falling back to node-gyp rebuild, after which GCC failed with:

fatal error: error writing to /tmp/cc*.s: Disk quota exceeded

The actionable cause was a per-user quota on /tmp, not globally exhausted disk space:

• /tmp: 15.7 GiB tmpfs mounted with usrquota
• User-owned data in /tmp: approximately 12.588 GiB, apparently at the per-user limit
• df still reported approximately 2.9 GiB globally available
• /home had approximately 66 GiB free

This is easy to misdiagnose if the updater checks only filesystem-wide free space.

The current error path retains the npm exit code and stdout/stderr lengths in PinnedRuntimeInstallError, then exposes the generic Could not prepare t3@… wrapper. The useful npm stderr never reaches the remote client.

Suggested improvements

1. Surface a bounded, sanitized installation diagnostic. Known failures could be classified as temporary-directory quota exceeded, filesystem out of space, missing compiler/toolchain, or permission denied. Raw npm output should remain private because it may contain paths, registry configuration, or credentials.
2. Preflight the temporary directory before starting a native build, accounting for per-user quota rather than only df/statfs free space.
3. Consider running pinned-runtime installation with a private TMPDIR under the T3 state directory or another user-writable location with sufficient capacity.
4. For unclassified errors, retain the redacted fallback but direct the operator to the relevant server/npm log.

No workaround was applied during triage.