Uh oh!
There was an error while loading. Please reload this page.
feat: add support for parallel kind processing with threads - #840
Conversation
2a48d4e to
05e295aCompareMsgspec does support freethreading in python: https://github.com/jcrist/msgspec/pull/877/files Initially thought we are planning to have both support for msgspec and voluptuous |
bhearsum
commented
Oct 24, 2025
Oh, that's wonderful to hear about msgspec! Supporting both should be fine as far as I'm concerned for this work; it just means I'll need to rebase this after the schema work lands. |
ofek
commented
Nov 24, 2025
msgspec support for 3.14 has been released https://github.com/jcrist/msgspec/releases/tag/0.20.0 |
2afc838 to
f4b9a78Comparef4b9a78 to
019cf90CompareThis will make a subsequent commit that adds support for a ThreadPoolExecutor cleaner.
Voluptuous contains thread-unsafe code; work around this by locking before calling into it.
Even with 3.14 free-threaded python, this is still a bit slower than multiprocessing on Linux, but it will allow us to start experimenting with it more, and may allow users on macOS and Windows to immediately see a speed-up.
hneiva
left a comment
There was a problem hiding this comment.
One non-blocking question
LGTM!
| # | ||
| # On all platforms, multiple threads can be enabled by setting | ||
| # TASKGRAPH_USE_THREADS in the environment. Taskgraph must be running | ||
| # from a free-threaded Python build to see any performance benefits. |
There was a problem hiding this comment.
Should we also add this info to taskgraph --help ?
There was a problem hiding this comment.
I think, because it's experimental, I'd rather avoid advertising it for the moment (aside from a mention in the changelog during the next release).
Uh oh!
There was an error while loading. Please reload this page.
This adds experimental, opt-in support for generating kinds in parallel across multiple threads. When used in conjunction with a free-threaded python it can be a significant speed-up in configuration with many expensive to generate kinds (eg: Gecko) where multiprocessing is not available.
In the short and medium term this should stay opt-in for a few reasons:
zstandard) are not available, which makes taskgraph more painful to install with this python.In the long term, I hope to get rid of multiprocessing support entirely once the above are no longer issues, because it imposes many limitations on how taskgraph generation can happen, how cached data can be shared, etc.
Ideally this PR would come with some multithreaded testing on Windows and macOS. Once #869 is done we ought to be able to add mac tests. Windows is trickier -- we don't have a compiler available on the workers, and zstandard (and probably other packages) don't have wheels available for free threaded python, which means we can't even install taskgraph in a free-threaded python virtualenv.
The code coverage failure is a bit misleading -- there's little in the way of new code here, and the new paths for the Linux side is covered with the multithread test job, which coverage isn't aware of.