Skip to content

fix(extensions): force UTF-8 stdio on Windows to stop UnicodeEncodeError crashes - #272

Merged
lightningpixel merged 1 commit into
lightningpixel:devfrom
rekcilyssup:fix/utf8-stdio-windows
Aug 20, 2026
Merged

fix(extensions): force UTF-8 stdio on Windows to stop UnicodeEncodeError crashes#272
lightningpixel merged 1 commit into
lightningpixel:devfrom
rekcilyssup:fix/utf8-stdio-windows

Conversation

@rekcilyssup

Copy link
Copy Markdown

Fixes#270 (UnicodeEncodeError during model auto-download) and the deadlock root cause described in #214.

Problem

On Windows, the embedded Python defaults stdout/stderr/stdin to the active console codepage (cp1252, cp932, ...). Two failure modes follow:

  1. Crash (reported in Windows: UnicodeEncodeError during model auto-download #270):BaseGenerator._auto_download() prints and ; on cp1252 stdout this raises UnicodeEncodeError: 'charmap' codec can't encode character '\u2192', aborting the model download for any extension using _auto_download() (e.g. triposplat).
  2. Deadlock (reported in 3D generation hangs forever on Japanese Windows: extension worker deadlocks when stderr reader thread dies with UnicodeDecodeError (cp932) #214): the extension worker emits UTF-8 (and mixed-encoding tqdm output). The parent's Popen(text=True) pipe readers decode with the OS locale, so the stderr reader thread dies with UnicodeDecodeError on Japanese Windows; the stderr pipe fills and generation hangs forever at 0% CPU.

Changes

  • api/services/generators/base.py — ASCII-ify the download progress print (-> / ...), the exact line from the Windows: UnicodeEncodeError during model auto-download #270 traceback.
  • api/services/generator_registry.py — ASCII-ify the reload print ( -> ...), same crash class in the API process.
  • api/services/stdio_utf8.py (new)ensure_utf8_stdio() reconfigures stdin/stdout/stderr to UTF-8 with errors="replace", no-op when unsupported.
  • api/main.py / api/runner.py — call ensure_utf8_stdio() at startup so both the FastAPI process and every extension worker emit UTF-8 regardless of locale.
  • api/services/extension_process.py — spawn workers with PYTHONUTF8=1 and Popen(..., encoding="utf-8", errors="replace") so parent-side pipe readers never die on non-UTF-8 locales (the verified fix from 3D generation hangs forever on Japanese Windows: extension worker deadlocks when stderr reader thread dies with UnicodeDecodeError (cp932) #214).
  • electron/main/process-runner.ts — set PYTHONUTF8=1 for process-extension spawns (same crash class).

Tests

  • New api/tests/test_stdio_utf8.py (3 tests): helper reconfigures all three streams, skips streams without reconfigure, swallows errors.
  • New api/tests/test_base_generator.py: regression test asserting the _auto_download print is ASCII-safe.
  • api/tests/test_extension_process.py: asserts _build_env() sets PYTHONUTF8=1.

All 6 new tests pass. Node suite: 95/95 pass. Pre-existing failures on origin/dev (2 registry tests + fastapi import in system Python) are unchanged by this PR.

…ror crashes
The embedded Python on Windows defaults stdout/stderr/stdin to the active
console codepage (cp1252, cp932, ...). Any Unicode print from Modly or an
extension generator (e.g. the arrow in _auto_download's progress line, tqdm
output) then crashes with UnicodeEncodeError, and reading UTF-8 worker
output under a legacy codepage can kill the stderr reader thread and
deadlock generation.
- ASCII-ify the unicode prints in BaseGenerator._auto_download and the
registry reload message so the reported crash cannot happen regardless
of encoding setup.
- Add services/stdio_utf8.ensure_utf8_stdio() and call it in the FastAPI
process (api/main.py) and the extension worker (api/runner.py) so all
Modly output is UTF-8 and matches the UTF-8 pipe readers on the
Electron side.
- Spawn extension workers with PYTHONUTF8=1 and explicit
encoding="utf-8", errors="replace" pipes so both ends agree on UTF-8
even on non-UTF-8 Windows locales.
- Force PYTHONUTF8=1 for process extensions spawned by the Electron
process runner.
Fixeslightningpixel#270; also addresses the deadlock root cause described in lightningpixel#214.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@rekcilyssup@lightningpixel