Make developer-specific test-harness paths configurable via environment variables

[kaijunli-infr]

Problem

Several test harnesses hard-code paths that only exist on the original author's machine, so on a fresh clone they cannot run and their unit tests fail before exercising any logic.

Scratch root /data/ycfeng/tmp, used for large intermediate outputs by:

• tests/performance/sim_walltime_scaling/sweep.py (DEFAULT_TEMP_ROOT, which also bounds FRONTIER_WALLTIME_TMPDIR)
• tests/e2e/moe_ep_baseline_replay.py (TMP_ROOT, used for path validation, child TMPDIR, and CLI defaults)
• tests/e2e/moe_ep_non_dummy_matrix.py (default --output-root for both matrix kinds)

Pinned Reference checkout /data/ycfeng/stepfun-performance-optimization/Frontier/worktrees/ref-afd-readonly, used by the PD-AF parity harness (tests/e2e/pd_af_parity/harness.py, reference_observer_bootstrap.py) and tests/integration/test_pdaf_reference_lifecycle_observer.py.

Change

Two commits, one per path family. Both keep the historical path as the fallback when the variable is unset, so existing deployments are unchanged.

FRONTIER_TMP_ROOT (commit 1)

• Add tests/scratch_root.py with resolve_scratch_root(). The value must be absolute and is re-read on every call.
• The three harness modules resolve their root through it. FRONTIER_WALLTIME_TMPDIR keeps its semantics and must still resolve to a descendant of the (now relocatable) root. Rejection messages point at FRONTIER_TMP_ROOT as the remedy.
• Unit tests for the two affected harnesses point the root at pytest's tmp_path via an autouse fixture. New tests/unit/test_scratch_root.py covers the helper.

FRONTIER_PDAF_REFERENCE_REPO_ROOT (commit 2)

• Add tests/e2e/pd_af_parity/reference_repo_root.py with resolve_reference_repo_root().
• harness.py, reference_observer_bootstrap.py, and the integration test resolve REFERENCE_REPO_ROOT through it once at import, so the many call sites that treat it as a stable pin keep working. Only the location is configurable; the pinned git HEAD and source SHA-256 checks are unchanged. The bootstrap's mismatch error names the variable.
• test_pdaf_parity_harness_wave1.py builds its producer payload from the resolved root instead of a literal. New tests/unit/test_pdaf_reference_repo_root.py covers the helper.

AGENTS.md documents both variables under Tests and gets a Modification History row.

Verification

Fresh clone on a machine without /data/ycfeng (environment.yml + pip install -e ".[test]", plus torch/matplotlib), running tests/unit tests/integration:

Scope	Before	After
test_sim_walltime_scaling_sweep.py + test_moe_ep_baseline_replay.py	13 failed, 96 passed	0 failed, 110 passed
Overall	87 failed, 6 errors	74 failed, 6 errors

With FRONTIER_PDAF_REFERENCE_REPO_ROOT pointed at an existing plain directory, 17 of the 51 bootstrap unit tests additionally pass; the remaining 34 (and the 5 integration errors) need a real checkout at the pinned commit dcb1cc8e, which is not in this repository's public history, so they cannot pass anywhere without that private worktree. That is inherent to the identity checks and unchanged by this PR.

Other remaining failures pre-exist on main and are unrelated: docs-contract tests referencing a missing tests/debug fixture directory, and a few tests needing the profiling env.

Also verified manually: with FRONTIER_TMP_ROOT set, sweep._resolve_temp_root() returns the override and rejects FRONTIER_WALLTIME_TMPDIR=/tmp; with both variables unset, both helpers return their historical paths.

Out of scope

The /local/ycfeng/... interpreter defaults in the GPU profiling shell scripts under tests/analysis, tests/e2e/operator_parity, and tests/performance are not touched here.

[fwyc0573]

Hi Kevin, thanks for your PR. Here are some review comments I put together with the help of AI—feel free to take a look. If you have any follow‑up updates, that would be even better. Appreciate it!

Comment 1

Severity: BLOCK
File: tests/e2e/pd_af_parity/reference_observer_bootstrap.py:18-21

This import breaks the PD-AF reference integration path.

The integration test sets the child process PYTHONPATH to the pinned Reference checkout (tests/integration/ test_pdaf_reference_lifecycle_observer.py:889 and :1050). That checkout does not contain tests.e2e.pd_af_parity.reference_repo_root, so the bootstrap
exits before argument parsing with:

ModuleNotFoundError: No module named 'tests.e2e.pd_af_parity'

Please make the bootstrap self-contained under a Reference-only PYTHONPATH. Move the environment-variable parsing into the bootstrap or load a helper by
absolute file path without importing the current repository's tests package. Do not add the current repository to PYTHONPATH, because its frontier
package could override the pinned Reference runtime.

Please add a subprocess regression test that runs the observer-enabled Reference path with the same PYTHONPATH setup used by the integration test.

Comment 2

Severity: MEDIUM
File: tests/e2e/pd_af_parity/reference_repo_root.py:27-35

The Reference root is represented inconsistently across processes.

This resolver returns the raw environment-variable path. The bootstrap and integration driver call Path.resolve() before writing the path to the
sidecar, while the harness compares the sidecar value with the unresolved REFERENCE_REPO_ROOT string.

A valid path containing .. or a symlink can therefore produce a false parity mismatch. For example, the sidecar may contain /workspace/reference,
while the harness expects /workspace/link/../reference.

Please use one canonical representation everywhere. Either canonicalize the path in the shared resolver or call .resolve(strict=True) before every
producer and validator comparison.

Please add tests for:

• an override containing ..;
• an override using a symlink;
• equality between the sidecar value and the harness validation value.

Comment 3

Severity: MEDIUM
File: tests/e2e/run_moe_ep_non_dummy_matrix.sh:15-19

This wrapper bypasses the new FRONTIER_TMP_ROOT setting.

The Python entry point uses resolve_scratch_root() only when --output-root is absent. This wrapper always passes --output-root, with /data/ycfeng/ tmp/frontier_non_dummy_matrix as its default. Setting only FRONTIER_TMP_ROOT therefore has no effect when this wrapper is used.

Please pass --output-root only when MATRIX_OUTPUT_ROOT is explicitly set. Otherwise, let the Python entry point resolve the default from
FRONTIER_TMP_ROOT.

Please verify both cases:

• with FRONTIER_TMP_ROOT set and MATRIX_OUTPUT_ROOT unset, output goes to <FRONTIER_TMP_ROOT>/frontier_non_dummy_matrix;
• with MATRIX_OUTPUT_ROOT set, the explicit output directory is preserved.

Merge Requirements

1. Fix the bootstrap import failure under the Reference-only PYTHONPATH.
2. Add a subprocess regression test for the observer-enabled Reference run.
3. Make Reference root canonicalization consistent across the resolver, bootstrap, integration test, sidecar, and harness.
4. Add .. and symlink path tests.
5. Update the MoE matrix shell wrapper so FRONTIER_TMP_ROOT controls its default output path.
6. Run the focused unit tests and the full PD-AF lifecycle integration test in a real git checkout.
7. Run:

bash -n tests/e2e/run_moe_ep_non_dummy_matrix.sh
git diff --check origin/main...origin/pr-29
pytest -q tests/integration/test_pdaf_reference_lifecycle_observer.py

[kaijunli-infr]

Thanks for the careful review. All three comments were valid, and I reproduced the blocker locally before fixing it. Addressed in two commits on top of the branch: d96b9fc (comments 1 and 2) and 5e1d1eb (comment 3).

Comment 1 (BLOCK): bootstrap import under a Reference-only PYTHONPATH. Confirmed: PYTHONPATH=<empty dir> python tests/e2e/pd_af_parity/reference_observer_bootstrap.py failed with ModuleNotFoundError: No module named 'tests' on the previous head, while main printed usage. The bootstrap now carries its own copy of the environment-variable resolver and imports nothing outside the standard library; the shared helper stays for the harness and the integration test, which run from this repository. The current repository is not added to PYTHONPATH. New tests in tests/unit/test_pdaf_reference_repo_root.py:

• an AST scan asserting the bootstrap imports only stdlib modules;
• a lockstep test asserting the bootstrap's resolver and the shared resolver agree on the variable name, the fallback, and their results for the unset, symlink, and .. cases, and both reject non-absolute values;
• two subprocess regression tests that launch the bootstrap by absolute path with cwd and PYTHONPATH set to a directory that does not contain this repository, mirroring the integration test's launch. One checks the bootstrap's own argument parser is what rejects an empty invocation; the other sets FRONTIER_PDAF_REFERENCE_REPO_ROOT and passes a mismatching --reference-repo-root, asserting the bootstrap's own mismatch error (which now names the variable) rather than an import failure.

Comment 2 (MEDIUM): canonical Reference root. Both resolvers now return resolve(strict=False) of the configured value, so the harness compares the same string the bootstrap and the integration driver write after their resolve(strict=True). Tests added for an override containing .., an override through a symlink, and equality between the sidecar value from _require_reference_root and the harness validation value under a symlink plus .. at once.

Comment 3 (MEDIUM): run_moe_ep_non_dummy_matrix.sh. The wrapper now forwards --output-root only when MATRIX_OUTPUT_ROOT is set; otherwise the Python entry point derives the default from FRONTIER_TMP_ROOT. New tests/unit/test_run_moe_ep_non_dummy_matrix_wrapper.py runs the wrapper with PYTHON_BIN pointed at a stub that echoes its arguments and checks both cases you listed (no --output-root and no literal developer path when unset; explicit directory preserved when set), plus bash -n. A test in test_moe_ep_non_dummy_matrix.py drives main() in preflight mode without --output-root and asserts the resolved root is <FRONTIER_TMP_ROOT>/frontier_non_dummy_matrix.

Merge requirements.
1 to 5: done as above.
7: bash -n tests/e2e/run_moe_ep_non_dummy_matrix.sh passes; git diff --check against main is clean; AGENTS.md updated for both variables.
6: I ran the focused unit tests and the full tests/unit tests/integration suite. The overall count is unchanged from the previous head (74 failed, 6 errors, all pre-existing on main), with the new tests passing. I cannot run pytest -q tests/integration/test_pdaf_reference_lifecycle_observer.py to completion here because it requires a checkout at the pinned commit dcb1cc8e, which is not in this repository's public history. Could you run that one on a machine that has the Reference worktree? Everything else in the launch path (script-by-path, Reference-only PYTHONPATH, foreign cwd) is covered by the new subprocess tests.