Conversation
|
Hi, thanks for flagging this issue. I understand this could happen when users on a CUDA-enabled PyTorch build are doing operations on both their accelerators and their CPU. It seems like a reasonable change to me. Just a few nits before I'm ready to merge. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Hello @matthewdouglas Thank you so much for your review. Whenever possible for you, would appreciate if you could take another look at the pr. Thank you |
Description
The fused CPU 4-bit GEMM loader knows it is registering a CPU kernel, but it let
kernelsinfer the backend from the installed PyTorch build. With a CUDA PyTorch wheel, that selects CUDA even thoughkernels-community/quantization-bitsandbytespublishes a CPU build, so the fused kernel is rejected and bitsandbytes falls back after logging a warning.Pass
backend="cpu"explicitly when loading the Hub kernel. Explicit backend selection requireskernels >= 0.13.0; older or missing versions continue to use the existing best-effort warning and fallback behavior.Tests
.venv/bin/pytest tests/test_ops.py::Test4bitBlockwiseQuantOps::test_gemv_4bit -k cpu -v --tb=short(48 passed, 12 skipped, 60 deselected)pre-commit run --all-files(all hooks passed)Fixes #2064
AI assistance: This change was implemented with OpenAI Codex and independently reviewed and validated before submission.