Skip to content

[finding] --union-into pushes an ABSOLUTE packages.items[].path while turbo ls emits repo-relative ones, so the document it hands the shard partitioner is heterogeneous #10056

Description

@os-steve

Found while auditing every field of the turbo ls document for #10046 (that card fixed the sibling defect in the same writer, packages.count). Filed unassigned. Inert today, and unlike the count case the correct fix is a judgment call rather than mechanical — which is why it is a separate card and was deliberately not folded into #10046's PR.

The divergence

turbo ls --output=json emits packages.items[].path as a repo-relative path. Measured on turbo 2.10.10 against this repo, all 77 packages:

$ pnpm exec turbo ls --output=json | node -e "...
items with ABSOLUTE path: 0 of 77
sample paths: packages/apps/account, packages/cli, packages/client

scripts/check-cross-package-test-inputs.mjs's unionInto() appends an absolute one:

items.push({ name,path: join(REPO_ROOT,dir)});

Measured with a changed-file list of exactly scripts/sync-template-versions.mjs:

{"name":"@objectstack/spec","path":"/home/user/objectstack-issue-10046/packages/spec"}
{"name":"create-objectstack","path":"/home/user/objectstack-issue-10046/packages/create-objectstack"}

So a single document can carry both conventions in one array — turbo's entries relative, the unioned entries absolute.

Why it is inert today

The only consumer is scripts/partition-test-shards.mjs, which does readdirSync(it.path) via countTestFiles(). That resolves a relative path against process.cwd(), and CI runs the script from the repo root, so both conventions land on the same directory. Measured, same package, same tree:

relative (turbo's convention): shard 1/1: 1/1 packages, weight 414
absolute (union's convention): shard 1/1: 1/1 packages, weight 414

Why it is still worth closing

countTestFiles() returns 0 for a path it cannot read — a deliberate, documented choice (// package path missing locally -- weight 0, still assigned). That is what keeps this quiet: a path resolved against the wrong base does not throw, it produces weight 0, and the LPT partitioner absorbs a zero weight without complaint. Measured — same relative-path document, run from a different cwd:

relative, cwd = /: shard 1/1: 1/1 packages, weight 0

The package is still assigned, still tested; only the load balancing silently degrades. So the failure mode of this class is not a red step, it is a shard matrix that quietly stops balancing.

A future consumer is the sharper risk: anything reaching for the obvious join(REPO_ROOT, it.path) — correct for every entry turbo wrote — produces a garbage path for exactly the unioned entries, which are the cross-package scans #7802 exists to keep running.

Why this is not a one-line fix

Both normalisations are defensible and they trade against each other:

  • Normalise the union to relative — the document becomes a faithful turbo ls payload, which is what partition-test-shards.mjs's assert-the-shape-loudly posture wants. But relative paths are cwd-dependent, and the consumer's own weight function fails silently (weight 0, above) when the cwd assumption breaks.
  • Normalise everything to absolute — robust against cwd, but the document is then no longer what turbo ls emits, and it would need rewriting turbo's own entries, i.e. mutating fields the union step does not author.

There is also a third option: leave the data alone and have the consumer resolve it.path against the repo root explicitly, which fixes the cwd fragility for both conventions and makes the heterogeneity harmless rather than merely inert.

⚠️ Whichever is chosen, note the entry-gate consequence: it.path feeds countTestFiles() feeds the package weight feeds shard placement. A change here can move packages between shards, so it is not a comment/data-shape change — which is precisely why #10046's PR left it alone (that card's claim declared no change to which packages get sharded).

Refs: #10046 (sibling defect in the same writer, and where this was measured), #7802 (the failure family the union step exists for).


Generated by Claude Code

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions