fix(extensions): force UTF-8 stdio on Windows to stop UnicodeEncodeError crashes - #272
Merged
lightningpixel merged 1 commit intoAug 20, 2026
Conversation
…ror crashes The embedded Python on Windows defaults stdout/stderr/stdin to the active console codepage (cp1252, cp932, ...). Any Unicode print from Modly or an extension generator (e.g. the arrow in _auto_download's progress line, tqdm output) then crashes with UnicodeEncodeError, and reading UTF-8 worker output under a legacy codepage can kill the stderr reader thread and deadlock generation. - ASCII-ify the unicode prints in BaseGenerator._auto_download and the registry reload message so the reported crash cannot happen regardless of encoding setup. - Add services/stdio_utf8.ensure_utf8_stdio() and call it in the FastAPI process (api/main.py) and the extension worker (api/runner.py) so all Modly output is UTF-8 and matches the UTF-8 pipe readers on the Electron side. - Spawn extension workers with PYTHONUTF8=1 and explicit encoding="utf-8", errors="replace" pipes so both ends agree on UTF-8 even on non-UTF-8 Windows locales. - Force PYTHONUTF8=1 for process extensions spawned by the Electron process runner. Fixeslightningpixel#270; also addresses the deadlock root cause described in lightningpixel#214.
lightningpixel
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes#270 (UnicodeEncodeError during model auto-download) and the deadlock root cause described in #214.
Problem
On Windows, the embedded Python defaults
stdout/stderr/stdinto the active console codepage (cp1252,cp932, ...). Two failure modes follow:BaseGenerator._auto_download()prints→and…; oncp1252stdout this raisesUnicodeEncodeError: 'charmap' codec can't encode character '\u2192', aborting the model download for any extension using_auto_download()(e.g. triposplat).Popen(text=True)pipe readers decode with the OS locale, so the stderr reader thread dies withUnicodeDecodeErroron Japanese Windows; the stderr pipe fills and generation hangs forever at 0% CPU.Changes
api/services/generators/base.py— ASCII-ify the download progress print (->/...), the exact line from the Windows: UnicodeEncodeError during model auto-download #270 traceback.api/services/generator_registry.py— ASCII-ify the reload print (…->...), same crash class in the API process.api/services/stdio_utf8.py(new) —ensure_utf8_stdio()reconfigures stdin/stdout/stderr to UTF-8 witherrors="replace", no-op when unsupported.api/main.py/api/runner.py— callensure_utf8_stdio()at startup so both the FastAPI process and every extension worker emit UTF-8 regardless of locale.api/services/extension_process.py— spawn workers withPYTHONUTF8=1andPopen(..., encoding="utf-8", errors="replace")so parent-side pipe readers never die on non-UTF-8 locales (the verified fix from 3D generation hangs forever on Japanese Windows: extension worker deadlocks when stderr reader thread dies with UnicodeDecodeError (cp932) #214).electron/main/process-runner.ts— setPYTHONUTF8=1for process-extension spawns (same crash class).Tests
api/tests/test_stdio_utf8.py(3 tests): helper reconfigures all three streams, skips streams withoutreconfigure, swallows errors.api/tests/test_base_generator.py: regression test asserting the_auto_downloadprint is ASCII-safe.api/tests/test_extension_process.py: asserts_build_env()setsPYTHONUTF8=1.All 6 new tests pass. Node suite: 95/95 pass. Pre-existing failures on
origin/dev(2 registry tests +fastapiimport in system Python) are unchanged by this PR.