deterministic cpu association - #2
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to make GaMMA’s multi-process association reproducible and CPU-bounded by enforcing deterministic clustering/association behavior and preventing CPU oversubscription from native threadpools.
Changes:
- Adds deterministic controls in association: DBSCAN forced single-threaded, deterministic per-cluster seeding, and fixed-order merging of results.
- Clamps association worker count to available CPUs (respecting CPU affinity) and to the number of clusters.
- Adds
threadpoolctldependency and initializes per-worker native threadpool limits to 1 thread.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| setup.py | Adds threadpoolctl dependency and reformats requirements list. |
| gamma/utils.py | Implements deterministic clustering/association behavior, CPU/cluster worker clamping, and per-worker native threadpool limiting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Addressed the review suggestions and resolved the conversations. The PR is ready for re-review. |
Pamicoding
left a comment
There was a problem hiding this comment.
Hi @hsian-coding bro, I think I should submit review in formal way that can let AI to know there are still have some issues need to solve. Issues include three comments we've closed (R252-R269, R273, R278-282), since close the comments did not solve it directly, I might still need another push from you to solve this, many thanks bro.
This PR makes GaMMA’s multi-process association CPU-bounded and reproducible. It clamps worker counts to available CPUs and clusters, runs DBSCAN single-threaded for stable labels, applies deterministic per-cluster seeds, and merges results in a fixed order so event IDs and pick assignments remain consistent across worker counts.
It also limits native math libraries to one thread per association worker to prevent CPU oversubscription.