Skip to content

Add an opt-in timeout for VLLMClient metadata requests - #7219

Open
030611 wants to merge 1 commit into
huggingface:mainfrom
030611:fix/metadata-read-timeout
Open

030611 wants to merge 1 commit into
huggingface:mainfrom
030611:fix/metadata-read-timeout

Conversation

@030611

@030611 030611 commented Sep 15, 2026

Copy link
Copy Markdown

What does this PR do?

Adds an optional metadata_timeout to VLLMClient for model discovery and world-size requests. A server that accepts either request but never responds now exhausts the existing read retries when a timeout is configured. The default remains unset.

This follows the metadata-only scope discussed in #6973 and leaves health probes (#6977), generation and weight-update timeout/retry policy separate. The value applies to connection establishment and read inactivity per attempt, rather than imposing an overall deadline.

Refs #6973. Discussion: #6973 (comment)

Local validation uses the complete production client and unchanged new test definitions with isolated collection. A localhost HTTP server exercises healthy and stalled responses, the existing retry count, and the separate POST timeout policy. GPU, NCCL and training integration were not run. Full test-file collection requires dependencies missing from this local environment.

Before submitting

AI writing disclosure

  • No AI usage.
  • AI-assisted: written and reviewed by a human.
  • AI-generated: the PR was mostly or fully generated by an AI tool.

Codex prepared the implementation and tests. I reviewed the changes before submitting.

Validated on upstream 0b4b33d: 5 isolated-collection HTTP tests passed, and all changed-file pre-commit hooks passed. The full upstream test file and GPU/NCCL/training integrations remain unverified.


Note

Low Risk
Opt-in behavior with default unchanged; scope limited to two metadata GET paths and covered by new HTTP tests.

Overview
Adds an optional metadata_timeout on VLLMClient so model discovery (/v1/models) and get_world_size (/get_world_size) can fail instead of hanging when the server accepts but never responds. The value is passed as the requests timeout on those GETs only; default remains no timeout. Health checks, generation, and weight-update POSTs are unchanged.

Regression tests use a threaded localhost HTTP stub: healthy metadata with and without a timeout, stalled responses that exhaust read retries (six attempts), and confirmation that _post does not receive the metadata timeout.

Reviewed by Cursor Bugbot for commit a82eee0. Bugbot is set up for automated code reviews on this repo. Configure here.

Signed-off-by: Errant <2843409461@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant