Skip to content

Enable bitsandbytes quantization on AMD GPUs that use warp size 32 #27307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
+10 −4
Open

Enable bitsandbytes quantization on AMD GPUs that use warp size 32 #27307

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e94363c
Enable bitsandbytes quantization on warp size 32 AMD GPUs
sstamenk Oct 21, 2025
494b4d6
Enable bitsandbytes unit tests on Radeon
sstamenk Nov 10, 2025
0f0fa5d
Update comment
sstamenk Nov 10, 2025
26342aa
Update test_bitsandbytes.py
sstamenk Nov 16, 2025
a39949a
Update rocm.py
sstamenk Nov 16, 2025
9fe767d
Refactor import logic
sstamenk Nov 17, 2025
6230a5d
Fix indentation
sstamenk Nov 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions tests/models/quantization/test_bitsandbytes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,13 @@
from ...utils import compare_two_settings, multi_gpu_test
from ..utils import check_embeddings_close, check_logprobs_close

pytestmark = pytest.mark.skipif(
current_platform.is_rocm(),
reason="bitsandbytes quantization not supported on ROCm (CUDA-only kernels)",
)
if current_platform.is_rocm():
from vllm.platforms.rocm import on_gfx9

pytestmark = pytest.mark.skipif(
on_gfx9(),
reason="bitsandbytes quantization not supported on gfx9 (warp size 64 limitation)",

Check failure on line 22 in tests/models/quantization/test_bitsandbytes.py

View workflow job for this annotation

GitHub Actions / pre-commit

Ruff (E501) 

tests/models/quantization/test_bitsandbytes.py:22:89: E501 Line too long (91 > 88)
)

models_4bit_to_test = [
("facebook/opt-125m", "quantize opt model inflight"),
Expand Down
3 changes: 3 additions & 0 deletions vllm/platforms/rocm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,9 @@ class RocmPlatform(Platform):
"petit_nvfp4",
"torchao",
]
# bitsandbytes quantization not supported on gfx9 (warp size 64 limitation)
if not on_gfx9():
supported_quantization += ["bitsandbytes"]

@classmethod
def get_vit_attn_backend(
Expand Down
Loading