Skip to content

fix: use hickory-dns to avoid musl getaddrinfo failures in Kubernetes - #2054

Closed
bsquizz wants to merge 1 commit into
NVIDIA:mainfrom
bsquizz:fix/musl-dns-hickory-resolver
Closed

fix: use hickory-dns to avoid musl getaddrinfo failures in Kubernetes#2054
bsquizz wants to merge 1 commit into
NVIDIA:mainfrom
bsquizz:fix/musl-dns-hickory-resolver

Conversation

@bsquizz

Copy link
Copy Markdown

Summary

Enable the hickory-dns feature on reqwest so the supervisor's router uses a pure-Rust DNS resolver instead of musl's getaddrinfo.

Problem

The supervisor is statically linked with musl libc. musl's getaddrinfo sends A and AAAA DNS queries simultaneously on a single UDP socket. With Kubernetes' default ndots:5 and 5 search domains, external FQDNs like aiplatform.googleapis.com (3 dots < 5) get expanded through all search domains first — 12+ concurrent queries whose responses arrive out of order, causing musl to return zero usable addresses. This manifests as 503 "inference service unavailable" on every inference call to external providers (Vertex AI, Anthropic API, etc.).

Fix

One-line change in workspace Cargo.toml:

-reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots", "hickory-dns"] }

hickory-dns (formerly trust-dns) is a pure-Rust async DNS resolver. It bypasses musl's getaddrinfo entirely, eliminating the concurrent A+AAAA query issue regardless of ndots configuration. This also aligns with the supervisor's static-binary design — no libc dependency for DNS resolution.

Verification

  • cargo check -p openshell-sandbox — compiles cleanly
  • cargo check -p openshell-router — compiles cleanly

Fixes#2053

The supervisor binary is statically linked with musl libc, whose
getaddrinfo sends A+AAAA queries simultaneously on a single UDP socket.
With Kubernetes' default ndots:5 and multiple search domains, external
FQDNs get expanded through all search domains first, generating 12+
concurrent queries whose responses arrive out of order — causing musl
to return zero usable addresses and the router to emit 503 "inference
service unavailable".
Enable the `hickory-dns` feature on reqwest so all HTTP clients use a
pure-Rust DNS resolver instead of musl's getaddrinfo. This bypasses the
concurrent A+AAAA issue entirely and aligns with the supervisor's
static-binary design (no libc dependency for DNS).
FixesNVIDIA#2053
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@copy-pr-bot

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

Thank you for your interest in contributing to OpenShell, @bsquizz.

This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer.

To get vouched:

  1. Open a Vouch Request discussion.
  2. Describe what you want to change and why.
  3. Write in your own words — do not have an AI generate the request.
  4. A maintainer will comment /vouch if approved.
  5. Once vouched, open a new PR (preferred) or reopen this one after a few minutes.

See CONTRIBUTING.md for details.

@github-actions

github-actionsBot commented Jun 30, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@bsquizz

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2054 is at {"messa while the PR head is 9a2b333. A maintainer needs to comment /ok to test 9a2b333f68b53f2f9d8c8ef5e0bad203f999415c to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test 9a2b333

@mrunalp

Copy link
Copy Markdown
Collaborator

@bsquizz Can we check if #2682 addresses this?

@johntmyers

Copy link
Copy Markdown
Collaborator

I will close for now, but we can re-open if needed.

@bsquizz

Copy link
Copy Markdown
Author

Tested today with v0.0.109 running in a kind cluster. I can see options ndots:5 in /etc/resolv.conf on the sandbox and inference routing for a vertex provider is successful. Seems like the problem is resolved.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2eRequires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Supervisor's musl DNS resolver fails for external domains in Kubernetes (ndots:5)

4 participants

@bsquizz@TaylorMutch@mrunalp@johntmyers