Uh oh!
There was an error while loading. Please reload this page.
fix: use hickory-dns to avoid musl getaddrinfo failures in Kubernetes - #2054
fix: use hickory-dns to avoid musl getaddrinfo failures in Kubernetes#2054bsquizz wants to merge 1 commit into
Conversation
The supervisor binary is statically linked with musl libc, whose getaddrinfo sends A+AAAA queries simultaneously on a single UDP socket. With Kubernetes' default ndots:5 and multiple search domains, external FQDNs get expanded through all search domains first, generating 12+ concurrent queries whose responses arrive out of order — causing musl to return zero usable addresses and the router to emit 503 "inference service unavailable". Enable the `hickory-dns` feature on reqwest so all HTTP clients use a pure-Rust DNS resolver instead of musl's getaddrinfo. This bypasses the concurrent A+AAAA issue entirely and aligns with the supervisor's static-binary design (no libc dependency for DNS). FixesNVIDIA#2053 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Thank you for your interest in contributing to OpenShell, @bsquizz. This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer. To get vouched:
See CONTRIBUTING.md for details. |
All contributors have signed the DCO ✍️ ✅ |
bsquizz
commented
Jul 11, 2026
I have read the DCO document and I hereby sign the DCO. |
Label |
TaylorMutch
commented
Jul 13, 2026
/ok to test 9a2b333 |
mrunalp
commented
Aug 13, 2026
johntmyers
commented
Aug 17, 2026
I will close for now, but we can re-open if needed. |
bsquizz
commented
Aug 19, 2026
Tested today with v0.0.109 running in a kind cluster. I can see |
Summary
Enable the
hickory-dnsfeature on reqwest so the supervisor's router uses a pure-Rust DNS resolver instead of musl'sgetaddrinfo.Problem
The supervisor is statically linked with musl libc. musl's
getaddrinfosends A and AAAA DNS queries simultaneously on a single UDP socket. With Kubernetes' defaultndots:5and 5 search domains, external FQDNs likeaiplatform.googleapis.com(3 dots < 5) get expanded through all search domains first — 12+ concurrent queries whose responses arrive out of order, causing musl to return zero usable addresses. This manifests as 503"inference service unavailable"on every inference call to external providers (Vertex AI, Anthropic API, etc.).Fix
One-line change in workspace
Cargo.toml:hickory-dns(formerly trust-dns) is a pure-Rust async DNS resolver. It bypasses musl'sgetaddrinfoentirely, eliminating the concurrent A+AAAA query issue regardless ofndotsconfiguration. This also aligns with the supervisor's static-binary design — no libc dependency for DNS resolution.Verification
cargo check -p openshell-sandbox— compiles cleanlycargo check -p openshell-router— compiles cleanlyFixes#2053