feat(qwen3): bundle Qwen3 tokenizer for offline single-file/GGUF encoders

[Pfannkuchensack]

Summary

Single-file (safetensors) and GGUF Qwen3 encoder checkpoints used by Anima
(0.6B) and Z-Image (4B/8B) ship weights only — no tokenizer files. The loader
pulled the tokenizer from Qwen/Qwen3-4B on HuggingFace, which fails offline /
airgapped and whenever the HF cache is not persisted (e.g. Docker without a
cache volume).

Vendor the self-contained Qwen3 fast tokenizer (Apache-2.0, from Qwen/Qwen3-4B)
in the package and load it locally, mirroring the bundled T5-XXL tokenizer
(#9244). The Qwen3 BPE tokenizer is identical across the 0.6B/4B/8B variants, so
a single copy serves every Qwen3 encoder. Removes the HuggingFace download path
from both the checkpoint and GGUF loaders.

Changes

• Add vendored tokenizer under invokeai/backend/qwen3/tokenizer/ (tokenizer.json, tokenizer_config.json, special_tokens_map.json).
• Add load_bundled_qwen3_tokenizer() helper (invokeai/backend/qwen3/qwen3_tokenizer.py), mirroring anima/t5_tokenizer.py.
• Point Qwen3EncoderCheckpointLoader and Qwen3EncoderGGUFLoader at the bundled tokenizer; drop the dead DEFAULT_TOKENIZER_SOURCE HF-download fallback.
• Register the tokenizer files as package data in pyproject.toml so they ship in the wheel.

Related Issues / Discussions

Follows the same approach as #9244 (bundled T5-XXL tokenizer for Anima). No linked issue.

A follow-up will apply the same fix to the Qwen2.5-VL encoder (Qwen-Image / Qwen-Image-Edit), which has the identical single-file problem for its tokenizer, config, and image processor.

QA Instructions

1. Automated: run the new offline test — fully passes without network:

   HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 pytest tests/backend/qwen3/test_qwen3_tokenizer.py
   

2. Manual (offline single-file encoder):

   • Remove/rename the HF cache for the repo to force the offline path: ~/.cache/huggingface/hub/models--Qwen--Qwen3-4B.
   • Start Invoke with no network (or HF_HUB_OFFLINE=1).
   • Install a single-file Qwen3 encoder (Anima 0.6B, or a Z-Image 4B/8B .safetensors) and run a generation.
   • Expected: prompt encoding succeeds with no network request; previously this raised an OSError / HF download error.

3. Regression: confirm a diffusers-folder Qwen3 encoder (with its own tokenizer/) and the GGUF encoder still load and generate correctly.

Merge Plan

Standard merge — no DB/schema changes. Note: adds ~11 MB of vendored tokenizer JSON to the package (same rationale/precedent as the bundled T5 tokenizer in #9244).

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — N/A (no frontend/redux changes)
• Documentation added / updated (if applicable) — N/A
• Updated What's New copy (if doing a release after this PR) — N/A

[lstein]

Pushed two fixes:

1. LFS check failure (2d5314e903)
The lfs checks job flagged invokeai/backend/qwen3/tokenizer/tokenizer.json (~10.9 MB) as over the 10 MB lfs-warning threshold. Git LFS isn't suitable here since the tokenizer must ship inside the wheel for offline/airgapped use (LFS pointer files wouldn't get packaged). Instead the file is now vendored gzip-compressed (tokenizer.json.gz, ~2 MB) and decompressed into a temp dir at load time — the fast tokenizer reads everything into memory at construction, so the temp files are cleaned up immediately after. pyproject.toml package-data updated to include tokenizer/*.json.gz.

2. Merge conflict with main (680d698e85)
Merged the latest main and resolved a pyproject.toml [tool.setuptools.package-data] conflict — main added a bundled t5 tokenizer entry while this branch added anima + qwen3 entries; kept all three.

[lstein]

Pushed a small follow-up cleanup (666723c216) from a self-review of the loader changes:

• Renamed _load_tokenizer_with_offline_fallback → _load_bundled_tokenizer in both Qwen3EncoderCheckpointLoader and Qwen3EncoderGGUFLoader. The method no longer has any offline fallback — it just returns the vendored tokenizer — so the old name was misleading.
• Fixed two stale call-site comments in z_image.py that still described the removed HuggingFace download / "try local cache first" path.

Purely comments + a private method rename — no behavior change. Left the identically-named method in qwen_image.py untouched, since that's the Qwen2.5-VL loader deferred to the follow-up PR. ruff is clean and all 5 offline tokenizer tests still pass.

[lstein]

Approving — tested the fully-offline path end to end (Z-Image Turbo GGUF Q4_K + Qwen3 4B GGUF Q6_K encoder, HF cache removed, HF_HUB_OFFLINE=1): generation succeeds with no network access.

I pushed a few follow-up commits while reviewing:

1. LFS check fix (2d5314e903) — tokenizer.json was ~11 MB, over the repo's 10 MB lfs-warning threshold, failing the lfs checks job. LFS is unsuitable here (the file must ship inside the wheel for offline use), so it's now vendored gzip-compressed (tokenizer.json.gz, ~2 MB) and decompressed into a temp dir at load time. pyproject.toml package-data updated accordingly.

2. Merge with main (680d698e85) — resolved a pyproject.toml package-data conflict (main added a t5 tokenizer entry; kept all three: anima/qwen3/t5).

3. Loader comment/name cleanup (666723c216) — renamed _load_tokenizer_with_offline_fallback → _load_bundled_tokenizer (no fallback remains) and fixed two stale call-site comments still referencing the removed HF-download path.

4. Restore chat_template (71a988faf3) — the real fix for a runtime crash. The vendored tokenizer_config.json shipped without the chat_template that Qwen/Qwen3-4B includes inline. The Z-Image encoder formats prompts via tokenizer.apply_chat_template(), which raised:

ValueError: Cannot use chat template functions because tokenizer.chat_template is not set ...

so GGUF/single-file encoders crashed at encode time. This was a regression from bundling — the old HF-download path pulled the full config. I restored the exact upstream Qwen3-4B chat_template (verified the config was otherwise byte-identical to the vendored one) and added a regression test so it can't be dropped again. All 6 tokenizer tests pass, ruff clean.

Note: I deliberately did not widen the encoder's except (AttributeError, TypeError) to swallow the ValueError — for Z-Image the chat template materially affects output quality, so shipping the correct template is the right fix rather than silently falling back to the raw prompt.