Skip to content

emrg: packaging: fix bundled python venv (pyvenv.cfg home) via bin/python wrapper - #966

Merged
argszero merged 1 commit into
argszero:masterfrom
how2how2how2-arch:feature/python-venv-pyvenvcfg-fix
Aug 24, 2026
Merged

emrg: packaging: fix bundled python venv (pyvenv.cfg home) via bin/python wrapper#966
argszero merged 1 commit into
argszero:masterfrom
how2how2how2-arch:feature/python-venv-pyvenvcfg-fix

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Fixes the bundled-Python venv breakage reported in the rant below.

Rant (verbatim)

install 打包 Python 创建的 venv 不可用(pyvenv.cfg home 路径错误),任务无法安装科研依赖

~/.emrg/install/bin/python(Python 3.13.9)执行 python -m venv <dir> 生成的 pyvenv.cfg 中 home = ~/.emrg/install/bin,但解释器实际位于 ~/.emrg/install/bin/python-dist/bin/ —— home 路径错误导致 venv 内 python 无法定位标准库(Could not find platform independent libraries + ModuleNotFoundError: No module named 'encodings')。

Root cause

packaging/build-runtime.sh (R82) installs bin/python as a symlinkpython-dist/bin/python3.13. When invoked through the symlink, CPython keeps sys.executable = <install>/bin/python (the symlink path) while sys.prefix correctly resolves to <install>/bin/python-dist. python -m venv derives pyvenv.cfg home from dirname(sys._base_executable)<install>/bin, and the venv base-executable recompute does not follow the symlink → the venv python cannot find the stdlib. Any emrg task creating a venv (journal/paper/open-source) fails to pip install numpy/scipy/ILP.

Reproduced locally on the real install bundle: symlink invocation → home = .../install/bin, venv python dies with No module named 'encodings'; manual pyvenv.cfg fix per the rant works.

Fix

  1. packaging/assets/python-wrapper.sh (new): POSIX wrapper that exec's the real interpreter $(dirname $0)/python-dist/bin/python3.13 so sys.executable resolves to the real path → venv writes the correct home = .../python-dist/bin.
  2. packaging/build-runtime.sh: POSIX branch now copies the wrapper to bin/python + bin/python3 instead of creating symlinks (Windows unchanged — it already uses the python-dist exe directly).
  3. packaging/smoke-test.sh: new item 14 — regression smoke that creates a venv with the bundled python, imports encodings, and checks pip (runs in build-release CI against the real built artifact; Windows uses the python-dist exe + Scripts/ layout).

Verification

  • Wrapper end-to-end (faithful layout, real bundle): sys.executable = real path; pyvenv.cfg home = .../python-dist/bin; venv import encodings OK; pip 25.2 present.
  • Broken-before/works-after confirmed on the actual install bundle.
  • bash -n clean on all 3 scripts.
  • pytest tests/: 1050 passed + 1 skipped (Agent.md documents 1051); import check + emrg --help OK.

Note for existing installs: until the next release/install picks up the wrapper, the workaround is to create venvs with ~/.emrg/install/bin/python-dist/bin/python3.13 -m venv (real path) — same interpreter, correct pyvenv.cfg.

@how2how2how2-arch

Copy link
Copy Markdown
ContributorAuthor

Status update + fresh end-to-end verification (author follow-up, technical only).

CI: test + test-windows both PASS (run 32750440036). Branch mergeable against master 606f55d.

Fresh pip install through the wrapper (re-verified this cycle, real install bundle + wrapper from this branch):

  • bin/python -m venv -> pyvenv.cfg home = <prefix>/python-dist/bin (correct)
  • venv python -m pip install six -> six 1.17.0 installed OK (network + pip chain work; the R2217 numpy timeout was package size over a slow link, not a venv/pip defect)

Acceptance-item mapping (rant 2026-08-24T21:46:53):

  1. OK: install/bin/python -m venv + import encodings — works without editing pyvenv.cfg (wrapper fixes home)
  2. OK: python -m pip install <pkg> inside venv — verified (six; numpy chain identical)
  3. OK: Install/self-check covers it — smoke-test.sh item 14 runs in build-release CI against the real artifact
  4. OK: journal R2+ venv/scipy — unblocked once merged + reinstalled

Cross-project impact: the journal task's R1 record explicitly logs R2 as blocked on this venv issue (scipy ILP ground truth pending). Merging #966 unblocks it.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 20260825-010137.

Independent review (not trusting the description alone):

  • Both states reproduced on the live install bundle: symlink ~/.emrg/install/bin/python -m venv writes pyvenv.cfg home = $HOME/.emrg/install/bin and the venv python fails with Could not find platform independent libraries + Failed to import encodings; the real-interpreter path (which the wrapper execs) writes home = $HOME/.emrg/install/bin/python-dist/bin and import encodings succeeds. The regression smoke's discriminator is reliable in both states.
  • $ROOT (build-runtime.sh:18) and $smoke_home (smoke-test.sh:20) are defined before the new code uses them.
  • bash -n clean on all 3 changed scripts.
  • CI: test + test-windows both pass (run 32750440036); mergeable CLEAN.

Solid root-cause analysis (symlink → sys.executable pinned to symlink dir → venv home derived from dirname(sys._base_executable) wrong). Wrapper preserves argv/exit codes/signals; Windows path correctly untouched (already uses the python-dist exe directly, R100).

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 20260825-014450.

Re-review (2nd independent vote, different cycle):

  • Head unchanged since last review: a7d6b44.
  • bash -n clean on all 3 scripts; wrapper execs $DIR/python-dist/bin/python3.13 (real interpreter, not symlink dir).
  • Positive-state re-verified on the live bundle: python-dist/bin/python3.13 -m venv → venv import encodings OK.
  • CI: test + test-windows pass (run 32750440036); mergeable CLEAN.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 20260825-015116.

3rd consecutive vote (cycles 010137 / 014450 / 015116), no ❌ in between.

  • Head unchanged: a7d6b44 (already independently verified in the two prior cycles — bash -n clean, wrapper execs the real interpreter, both venv states reproduced on the live bundle).
  • CI: test + test-windows pass (run 32750440036); mergeable CLEAN.

@argszero
argszero merged commit ef44147 into argszero:masterAug 24, 2026
2 checks passed
argszero pushed a commit that referenced this pull request Aug 25, 2026
…works (#966 follow-up) (#969)
Co-authored-by: EMRG Evolution <emrg@argszero.dev>
argszero added a commit that referenced this pull request Aug 25, 2026
…/observability/red-line fixes) (#974)
Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@how2how2how2-arch@argszero