Compatibility of SGLang / sgl-kernel with Blackwell GPUs (SM_120, CUDA 12.8)

[Vishalsng112]

I’m trying to run SGLang with Blackwell GPUs (SM_120) on CUDA 12.8, but I’m facing kernel compatibility issues.

System details:

GPU: NVIDIA Blackwell (SM_120)

Driver: (e.g., 570.169)

CUDA: 12.8

Python: (e.g., 3.11.x)

Install method: prebuilt wheels (sgl-kernel with blackwell support)

Error:
Exception: Capture cuda graph failed: RMSNorm failed with error code no kernel image is available for execution on the device (complete stack trace is given below.

Question:

Which versions of SGLang and sgl-kernel officially support SM_120 (Blackwell) + CUDA 12.8?

If support is not yet available, is there a roadmap / nightly build that I should try?

Stack trace:

[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  4.50it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  4.50it/s]

Capturing batches (bs=8 avail_mem=1.12 GB):   0%|                           | 0/4 [00:00<?, ?it/s]
[2025-08-23 23:01:10] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 394, in __init__
    self.capture()
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 499, in capture
    ) = self.capture_one_batch_size(bs, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 662, in capture_one_batch_size
    run_once()
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 650, in run_once
    logits_output_or_pp_proxy_tensors = forward(
                                        ^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 457, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 334, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 255, in forward
    hidden_states = self.input_layernorm(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/custom_op.py", line 59, in forward
    return self._forward_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/layers/layernorm.py", line 86, in forward_cuda
    out = rmsnorm(x, self.weight.data, self.variance_epsilon)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sgl_kernel/elementwise.py", line 45, in rmsnorm
    torch.ops.sgl_kernel.rmsnorm.default(out, input, weight, eps, enable_pdl)
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/torch/_ops.py", line 829, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: RMSNorm failed with error code no kernel image is available for execution on the device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2556, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 315, in __init__
    self.tp_worker = TpWorkerClass(
                     ^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 67, in __init__
    self.worker = TpModelWorker(
                  ^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 84, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 246, in __init__
    self.initialize(min_per_gpu_memory)
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 343, in initialize
    self.init_cuda_graphs()
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 1608, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/master/miniconda3/envs/sglang/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 396, in __init__
    raise Exception(
Exception: Capture cuda graph failed: RMSNorm failed with error code no kernel image is available for execution on the device
Possible solutions:
1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose 

Related resources

No response

[jhsmith409]

Try this:

services:
sglang:
image: lmsysorg/sglang:blackwell
container_name: sglang-server
ports:
- "30000:30000"
volumes:
- ./models:/models
- ./data:/data
environment:
- CUDA_VISIBLE_DEVICES=0
- TORCH_CUDA_ARCH_LIST=12.0
- HF_TOKEN= (fill in your token here)
shm_size: 32g
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
command: >
python3 -m sglang.launch_server
--model-path meta-llama/Llama-3.1-8B-Instruct
--tp 1
--attention-backend flashinfer
--host 0.0.0.0
--port 30000
restart: unless-stopped
stdin_open: true
tty: true

[CatchLee]

maybe you can manually downloading and installing sgl-kernel whl from https://github.com/sgl-project/whl/blob/484e8e47311e8cb58d210dcb8dcc8ae7c8d268d1/cu128/sgl-kernel/index.html#L57, works for my case.