Conversation
Signed-off-by: Errant <2843409461@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds an optional
metadata_timeouttoVLLMClientfor model discovery and world-size requests. A server that accepts either request but never responds now exhausts the existing read retries when a timeout is configured. The default remains unset.This follows the metadata-only scope discussed in #6973 and leaves health probes (#6977), generation and weight-update timeout/retry policy separate. The value applies to connection establishment and read inactivity per attempt, rather than imposing an overall deadline.
Refs #6973. Discussion: #6973 (comment)
Local validation uses the complete production client and unchanged new test definitions with isolated collection. A localhost HTTP server exercises healthy and stalled responses, the existing retry count, and the separate POST timeout policy. GPU, NCCL and training integration were not run. Full test-file collection requires dependencies missing from this local environment.
Before submitting
AI writing disclosure
Codex prepared the implementation and tests. I reviewed the changes before submitting.
Validated on upstream 0b4b33d: 5 isolated-collection HTTP tests passed, and all changed-file pre-commit hooks passed. The full upstream test file and GPU/NCCL/training integrations remain unverified.
Note
Low Risk
Opt-in behavior with default unchanged; scope limited to two metadata GET paths and covered by new HTTP tests.
Overview
Adds an optional
metadata_timeoutonVLLMClientso model discovery (/v1/models) andget_world_size(/get_world_size) can fail instead of hanging when the server accepts but never responds. The value is passed as the requeststimeouton those GETs only; default remains no timeout. Health checks, generation, and weight-update POSTs are unchanged.Regression tests use a threaded localhost HTTP stub: healthy metadata with and without a timeout, stalled responses that exhaust read retries (six attempts), and confirmation that
_postdoes not receive the metadata timeout.Reviewed by Cursor Bugbot for commit a82eee0. Bugbot is set up for automated code reviews on this repo. Configure here.