Prevent search workers from starving network requests - #6759
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 14a1317a44
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| fn search_thread_pool() -> &'static ThreadPoolWithPriority { | ||
| static SEARCH_THREAD_POOL: LazyLock<ThreadPoolWithPriority> = | ||
| LazyLock::new(|| ThreadPoolWithPriority::new("search", None)); | ||
| LazyLock::new(|| ThreadPoolWithPriority::new("search", Some(quickwit_common::num_cpus()))); |
There was a problem hiding this comment.
This makes QW_NUM_CPUS control the Rayon search-worker count, changing the operator-facing configuration contract and potentially altering search concurrency and resource sizing after an upgrade, but the commit updates no configuration or operational documentation. Document this new effect alongside the environment variable so deployments can size it with both Tokio and search workers in mind.
AGENTS.md reference: AGENTS.md:L23-L24
Useful? React with 👍 / 👎.
14a1317 to
25a7630
Compare
Size the search Rayon pool from Quickwit's configured CPU capacity instead of all CPUs visible on the node. In Kubernetes, searcher pods can request fewer CPUs than the node exposes, but Rayon otherwise creates a worker for every visible CPU. CPU-heavy searches can then saturate the node and delay load-reporting RPCs, causing searchers to be treated as unreachable.
25a7630 to
0fa4705
Compare
Summary
QW_NUM_CPUSis lowerContext
Kubernetes searcher pods can request fewer CPUs than their nodes expose. The Tokio runtimes respect
QW_NUM_CPUS, but the Rayon search pool previously used Rayon's default available parallelism and therefore created a worker for every visible node CPU. CPU-heavy searches could saturate the node and delay load-reporting RPCs, causing healthy searchers to be treated as unreachable.Validation
make fmtcargo check -p quickwit-search