Select type of GGUF quant

[slivka83]

Hello.

How to determine the type of quant when loading a GGUF model?

docker run \
    --gpus all \
    -p 30000:30000 \
    -v /models/bartowski/gemma-2-27b-it-GGUF:/model \
    --ipc=host \
    --name sg_gemma \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path /model \
        --host 0.0.0.0 \
        --port 30000 \
        --quantization gguf \
        --enable-p2p-check \
        --tp 2

[debkanchan]

Same problem here

I'm trying to run olmOCR-GGUF using sglang docker-compose. olmOCR-GGUF hosts multiple file for each quantisation. Usually with huggingface lbraries we can specify which file to load. But I was unable to specify it in sglang.