Skip to content

Reset the vLLM encoder cache after weight sync - #7200

Open
rmaguado wants to merge 1 commit into
huggingface:mainfrom
rmaguado:reset-mm-cache-after-weight-sync
Open

rmaguado wants to merge 1 commit into
huggingface:mainfrom
rmaguado:reset-mm-cache-after-weight-sync

Conversation

@rmaguado

@rmaguado rmaguado commented Sep 13, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes #7198

Previously, when syncing weights to vLLM, only the prefix cache is cleared, while the image encoder cache still has image embeddings computed by the previous policy.

Now, we reset the encoder cache in sync_weights, in both server and colocate mode.

Added reset_encoder_cache method to VLLMClient.

VLLMClient._post now returns an empty dict if the response has no content. Necessary for /reset_encoder_cache.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case. Opened vLLM encoder cache is not reset after weight sync #7198 for this (no discussion yet).
  • Did you make sure to update the documentation with your changes? N/A
  • Did you write any new necessary tests?

AI writing disclosure

We welcome the use of AI tools to help with contributions. For transparency and to help us improve our review process, please indicate the level of AI involvement in this PR.

  • No AI usage: the PR was written entirely by a human.
  • AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
  • AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.


Note

Medium Risk
Targets multimodal vLLM training correctness after weight updates; small HTTP client change affects any empty-body POST responses.

Overview
Clears stale vision encoder embeddings in vLLM after training weights are pushed, not just the KV prefix cache.

VLLMClient adds reset_encoder_cache() (POST /reset_encoder_cache) and makes _post return {} when the response body is empty so that endpoint does not break JSON parsing.

Weight sync in VLLMGeneration.sync_weights and OnlineDPOTrainer._move_model_to_vllm now calls encoder cache reset alongside reset_prefix_cache in both server (vllm_client) and colocate (llm_engine.reset_encoder_cache) modes.

Integration tests mirror test_reset_prefix_cache with test_reset_encoder_cache on the vLLM client/server suites.

Reviewed by Cursor Bugbot for commit ba72bd6. Bugbot is set up for automated code reviews on this repo. Configure here.

Previously, when syncing weights to vLLM we only reset the prefix cache,
so the encoder cache still has image embeddings computed by the
previous policy.

Now reset the encoder cache in sync_weights, in both server and
colocate modes.

Added `reset_encoder_cache` method to `VLLMClient`.

`VLLMClient._post` now returns an empty dict if the response has no
content. Necessary for `/reset_encoder_cache`.

Resolves: huggingface#7198
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vLLM encoder cache is not reset after weight sync

1 participant