Misc. bug: GLM-4.7-Flash Inference Much slower as compared to other A3B Models

[engrtipusultan]

Name and Version

bash  ./llama-server --version
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
version: 7822 (8f91ca5)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-bench, llama-server

Command line

./llama-bench -m /home/tipu/AI/models/ggml-org/GPT-OSS-20B/gpt-oss-20b-mxfp4.gguf -m /home/tipu/AI/models/ggml-org/Qwen3-Coder-30B-A3B/Qwen3-Coder-30B-A3B-Instruct-Q8_0.gguf -m /home/tipu/AI/models/unsloth/Nemotron-3-Nano/Nemotron-3-Nano-30B-A3B-Q8_0.gguf -m /home/tipu/AI/models/unsloth/GLM-4.7-Flash/GLM-4.7-Flash-Q8_0.gguf -ngl 100 --ubatch-size 512 --batch-size 2048 --mmap 0 -fa 0,1 -d 0,1024,2048,8096 -r 3 -dio 1

Problem description & steps to reproduce

GLM-4.7-Flash is just released. There are many issues raised around it. Model for me on my shared built is giving good output, but it TG and PP is much lower as compared to other A3B models.
Also as context grows its TG and PP drop to much.
Since MLA PR #18986 is merged I am not sure if there any indented PR to speed up the model.

I want to know that this slow down is due architecture of model or backends used by my machine need some update in implementation or something will be improved in llama.cpp to make inference of this model better.

I have mesa vulkan for GPU inference and AOCL BLAS installed for anything that goes to CPU.

AOCL: aocl-linux-gcc-5.2.0
VULKAN SDK: LunarG 1.4.335.0
VULKAN Driver: mesa-vulkan-drivers:amd64 25.3.3kisak1n amd64

Make options used:

cmake -S "$SOURCE_DIR" -B "$BUILD_DIR" -DGGML_NATIVE=ON -DSD_VULKAN=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=AOCL -DBLAS_INCLUDE_DIRS=/opt/AMD/aocl/aocl-linux-gcc-5.2.0/gcc/include -DCMAKE_PREFIX_PATH=/opt/AMD/aocl/aocl-linux-gcc-5.2.0/gcc
cmake --build "$BUILD_DIR" --config Release --clean-first --parallel $(nproc)

llama-bench build: 8f91ca5 (7822)

FA = off

Param	gpt-oss 20B MXFP4 MoE	nemotron_h_moe 31B.A3.5B Q8_0	qwen3moe 30B.A3B Q8_0	GLM4.7 Flash Q8_0
pp512	147.08 ± 1.49	113.02 ± 0.32	119.46 ± 0.18	102.81 ± 0.31
tg128	16.17 ± 0.00	12.05 ± 0.01	12.95 ± 0.01	10.77 ± 0.00
pp512 @ d1024	136.19 ± 1.73	111.24 ± 0.13	105.93 ± 0.34	86.65 ± 0.31
tg128 @ d1024	15.78 ± 0.03	11.84 ± 0.01	12.06 ± 0.06	7.29 ± 0.05
pp512 @ d2048	128.45 ± 1.21	108.86 ± 0.40	94.63 ± 0.51	73.20 ± 0.48
tg128 @ d2048	15.20 ± 0.03	11.50 ± 0.00	11.23 ± 0.00	5.28 ± 0.03
pp512 @ d8096	95.64 ± 0.76	98.47 ± 0.93	56.28 ± 0.18	38.71 ± 0.02
tg128 @ d8096	12.28 ± 0.01	9.17 ± 0.05	5.89 ± 0.05	2.19 ± 0.02

FA = on

params	gpt-oss 20B MXFP4 MoE	nemotron_h_moe 31B.A3.5B Q8_0	qwen3moe 30B.A3B Q8_0	GLM4.7 Flash Q8_0
pp512	146.69 ± 0.87	112.54 ± 0.65	114.26 ± 0.87	86.09 ± 0.12
tg128	16.64 ± 0.01	12.12 ± 0.01	13.39 ± 0.01	10.97 ± 0.01
pp512 @ d1024	132.76 ± 0.39	107.09 ± 0.32	77.43 ± 0.10	50.39 ± 0.10
tg128 @ d1024	16.36 ± 0.08	12.05 ± 0.01	12.29 ± 0.00	9.76 ± 0.01
pp512 @ d2048	120.38 ± 0.10	101.26 ± 0.28	55.47 ± 0.35	35.40 ± 0.02
tg128 @ d2048	16.11 ± 0.08	11.98 ± 0.00	11.66 ± 0.01	8.79 ± 0.00
pp512 @ d8096	77.32 ± 0.34	77.85 ± 0.48	20.76 ± 0.17	12.94 ± 0.01
tg128 @ d8096	14.91 ± 0.01	11.52 ± 0.00	8.92 ± 0.00	5.58 ± 0.00

First Bad Commit

None

Relevant log output

Logs

 tipu-dev-machine   ~/Development/GH/llama.cpp/build/bin  b7822 ≢  1                                                                      22:23:32 
 bash  ./llama-bench -m /home/tipu/AI/models/ggml-org/GPT-OSS-20B/gpt-oss-20b-mxfp4.gguf -m /home/tipu/AI/models/ggml-org/Qwen3-Coder-30B-A3B/Qwen3-Coder-30B-A3B-Instruct-Q8_0.gguf -m /home/tipu/AI/models/unsloth/Nemotron-3-Nano/Nemotron-3-Nano-30B-A3B-Q8_0.gguf -m /home/tipu/AI/models/unsloth/GLM-4.7-Flash/GLM-4.7-Flash-Q8_0.gguf -ngl 100 --ubatch-size 512 --batch-size 2048 --mmap 0 -fa 0,1 -d 0,1024,2048,8096 -r 3 -dio 1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | threads | fa | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | -: | --: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |           pp512 |        147.08 ± 1.49 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |           tg128 |         16.17 ± 0.00 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d1024 |        136.19 ± 1.73 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d1024 |         15.78 ± 0.03 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d2048 |        128.45 ± 1.21 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d2048 |         15.20 ± 0.03 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d8096 |         95.64 ± 0.76 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d8096 |         12.28 ± 0.01 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |           pp512 |        146.69 ± 0.87 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |           tg128 |         16.64 ± 0.01 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d1024 |        132.76 ± 0.39 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d1024 |         16.36 ± 0.08 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d2048 |        120.38 ± 0.10 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d2048 |         16.11 ± 0.08 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d8096 |         77.32 ± 0.34 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d8096 |         14.91 ± 0.01 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |           pp512 |        119.46 ± 0.18 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |           tg128 |         12.95 ± 0.01 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d1024 |        105.93 ± 0.34 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d1024 |         12.06 ± 0.06 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d2048 |         94.63 ± 0.51 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d2048 |         11.23 ± 0.00 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d8096 |         56.28 ± 0.18 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d8096 |          5.89 ± 0.05 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |           pp512 |        114.26 ± 0.87 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |           tg128 |         13.39 ± 0.01 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d1024 |         77.43 ± 0.10 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d1024 |         12.29 ± 0.00 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d2048 |         55.47 ± 0.35 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d2048 |         11.66 ± 0.01 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d8096 |         20.76 ± 0.17 |
| qwen3moe 30B.A3B Q8_0          |  30.25 GiB |    30.53 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d8096 |          8.92 ± 0.00 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |           pp512 |        113.02 ± 0.32 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |           tg128 |         12.05 ± 0.01 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d1024 |        111.24 ± 0.13 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d1024 |         11.84 ± 0.01 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d2048 |        108.86 ± 0.40 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d2048 |         11.50 ± 0.00 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d8096 |         98.47 ± 0.93 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d8096 |          9.17 ± 0.05 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |           pp512 |        112.54 ± 0.65 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |           tg128 |         12.12 ± 0.01 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d1024 |        107.09 ± 0.32 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d1024 |         12.05 ± 0.01 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d2048 |        101.26 ± 0.28 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d2048 |         11.98 ± 0.00 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d8096 |         77.85 ± 0.48 |
| nemotron_h_moe 31B.A3.5B Q8_0  |  31.27 GiB |    31.58 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d8096 |         11.52 ± 0.00 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |           pp512 |        102.81 ± 0.31 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |           tg128 |         10.77 ± 0.00 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d1024 |         86.65 ± 0.31 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d1024 |          7.29 ± 0.05 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d2048 |         73.20 ± 0.48 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d2048 |          5.28 ± 0.03 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |   pp512 @ d8096 |         38.71 ± 0.02 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  0 |   1 |   tg128 @ d8096 |          2.19 ± 0.02 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |           pp512 |         86.09 ± 0.12 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |           tg128 |         10.97 ± 0.01 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d1024 |         50.39 ± 0.10 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d1024 |          9.76 ± 0.01 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d2048 |         35.40 ± 0.02 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d2048 |          8.79 ± 0.00 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |   pp512 @ d8096 |         12.94 ± 0.01 |
| deepseek2 ?B Q8_0              |  29.65 GiB |    29.94 B | Vulkan,BLAS |       8 |  1 |   1 |   tg128 @ d8096 |          5.58 ± 0.00 |

build: 8f91ca54e (7822)

[engrtipusultan]

bash  GGML_VK_PERF_LOGGER=1 GGML_VK_PERF_LOGGER_FREQUENCY=99999 ./llama-bench -m /home/tipu/AI/models/unsloth/GLM-4.7-Flash/GLM-4.7-Flash-Q8_0.gguf -ngl 100 --ubatch-size 512 --batch-size 2048 --mmap 0 -fa 0,1 -d 0,1024,2048,8096 -r 3 -dio 1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	pp512	103.42 ± 0.19

---

Vulkan Timings

ADD: 192 x 302.625 us = 58104.1 us
CONCAT: 376 x 591.357 us = 222350 us
CONT: 188 x 513.28 us = 96496.6 us
GET_ROWS: 8 x 2.805 us = 22.44 us
GLU: 372 x 565.009 us = 210183 us
MUL: 180 x 1186.98 us = 213656 us
MULTI_ADD ADD: 184 x 671.648 us = 123583 us
MUL_MAT f16 m=512 n=512 k=512 batch=20: 188 x 4372.12 us = 821959 us (1226.74 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=576 batch=20: 188 x 6449.59 us = 1.21252e+06 us (935.649 GFLOPS/s)
MUL_MAT f32 m=64 n=512 k=2048: 180 x 395.59 us = 71206.3 us (339.202 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 8 x 16870 us = 134960 us (1272.65 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 360 x 2271.57 us = 817764 us (1417.72 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 4 x 16859 us = 67435.9 us (1273.73 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 180 x 2295.64 us = 413215 us (1402.74 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 188 x 7792.25 us = 1.46494e+06 us (1377.83 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 188 x 1927.83 us = 362431 us (1391.07 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 188 x 2200.1 us = 413618 us (912.697 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 188 x 2938.64 us = 552464 us (1369.31 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 188 x 891.391 us = 167582 us (1354.81 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 188 x 1210.21 us = 227520 us (1330.53 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 360 x 21602.7 us = 7.77698e+06 us (596.302 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 180 x 21633.7 us = 3.89406e+06 us (595.401 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 4 x 337.54 us = 1350.16 us (74.5323 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 8 x 290.91 us = 2327.28 us (86.4861 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 4 x 14.39 us = 57.56 us (18.2126 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 8 x 73.22 us = 585.76 us (85.9044 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 4 x 7177.2 us = 28708.8 us (88.3678 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 4 x 71.5 us = 286 us (87.9637 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 8 x 3.56 us = 28.48 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 372 x 213.474 us = 79412.6 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 188 x 65.927 us = 12394.4 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 188 x 82.194 us = 15452.6 us
ROPE: 376 x 101.535 us = 38177.3 us
SET_ROWS: 376 x 122.051 us = 45891.5 us
SOFT_MAX: 188 x 1609.62 us = 302609 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 184 x 15.047 us = 2768.8 us
Total time: 1.98531e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	tg128	10.34 ± 0.07

---

Vulkan Timings

ADD: 385 x 3.887 us = 1496.72 us
CONCAT: 36190 x 2.243 us = 81180 us
CONT: 18095 x 3.184 us = 57615.8 us
GET_ROWS: 770 x 3.616 us = 2784.64 us
GLU: 35805 x 2.529 us = 90579.2 us
MULTI_ADD ADD: 17710 x 4.847 us = 85845.3 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 504.497 us = 194231 us (83.1342 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 249.342 us = 4.41586e+06 us (84.0991 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 304.975 us = 5.40111e+06 us (82.4907 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 296.258 us = 1.04935e+07 us (84.9247 GFLOPS/s)
MUL_MAT_VEC f16 m=256 n=1 k=576 batch=20: 18095 x 56.208 us = 1.01709e+06 us (104.844 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=256 batch=20: 18095 x 62.797 us = 1.13633e+06 us (83.3253 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.409 us = 255195 us (18.1878 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 497.521 us = 383092 us (84.2833 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 71.731 us = 2.54073e+06 us (87.6871 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7271.72 us = 2.79961e+06 us (87.2192 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.605 us = 1.1973e+06 us (93.0304 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 249.56 us = 96080.9 us (84.0255 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 64.042 us = 1.15885e+06 us (81.7859 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 70.031 us = 1.26723e+06 us (56.0018 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 93.465 us = 1.69125e+06 us (84.0869 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.62 us = 336934 us (126.675 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.755 us = 882231 us (64.5047 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.075 us = 185625 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.862 us = 15607 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.413 us = 97959.5 us
ROPE: 36190 x 1.19 us = 43070.3 us
SET_ROWS: 36190 x 2.612 us = 94543.8 us
SOFT_MAX: 18095 x 6.324 us = 114444 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.427 us = 78403.5 us
Total time: 3.62158e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	pp512 @ d1024	85.65 ± 0.26

---

Vulkan Timings

ADD: 287 x 306.253 us = 87894.9 us
CONCAT: 564 x 612.135 us = 345244 us
CONT: 282 x 512.235 us = 144450 us
GET_ROWS: 10 x 2.764 us = 27.64 us
GLU: 556 x 573.475 us = 318852 us
MUL: 270 x 1205.7 us = 325541 us
MULTI_ADD ADD: 275 x 679.081 us = 186747 us
MUL_MAT f16 m=1024 n=512 k=576 batch=20: 47 x 11810.8 us = 555105 us (1021.87 GFLOPS/s)
MUL_MAT f16 m=1536 n=512 k=576 batch=20: 141 x 17516.4 us = 2.46981e+06 us (1033.53 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=1024 batch=20: 47 x 9182.18 us = 431562 us (1168.81 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=1536 batch=20: 141 x 13184.4 us = 1.859e+06 us (1221.2 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=512 batch=20: 94 x 4513.47 us = 424266 us (1188.33 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=576 batch=20: 94 x 6473.4 us = 608500 us (932.207 GFLOPS/s)
MUL_MAT f32 m=64 n=512 k=2048: 270 x 405.645 us = 109524 us (330.794 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 12 x 15169.9 us = 182038 us (1415.28 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 540 x 2293.14 us = 1.2383e+06 us (1404.38 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 6 x 15502.3 us = 93013.6 us (1385.2 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 270 x 2313.52 us = 624652 us (1391.89 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 282 x 7485.17 us = 2.11082e+06 us (1434.35 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 282 x 1926.98 us = 543407 us (1391.68 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 282 x 2204.16 us = 621574 us (911.014 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 282 x 2927.57 us = 825575 us (1374.49 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 282 x 876.195 us = 247087 us (1378.31 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 282 x 1211.64 us = 341682 us (1328.96 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 540 x 21333.1 us = 1.15199e+07 us (603.838 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 270 x 21588.2 us = 5.82881e+06 us (596.655 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 5 x 338.616 us = 1693.08 us (74.2955 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 10 x 291.444 us = 2914.44 us (86.3277 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 5 x 14.264 us = 71.32 us (18.3735 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 10 x 73.628 us = 736.28 us (85.4284 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 5 x 7209.45 us = 36047.2 us (87.9726 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 5 x 68.984 us = 344.92 us (91.172 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 10 x 3.636 us = 36.36 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 558 x 213.144 us = 118935 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 282 x 65.565 us = 18489.4 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 282 x 81.472 us = 22975.1 us
ROPE: 564 x 109.371 us = 61685.8 us
SET_ROWS: 564 x 124.072 us = 69977.1 us
SOFT_MAX: 282 x 3371.08 us = 950644 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 275 x 15.255 us = 4195.28 us
Total time: 3.33321e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	tg128 @ d1024	7.12 ± 0.00

---

Vulkan Timings

ADD: 385 x 3.903 us = 1502.88 us
CONCAT: 36190 x 3.915 us = 141691 us
CONT: 18095 x 3.209 us = 58080.4 us
GET_ROWS: 770 x 3.626 us = 2792.36 us
GLU: 35805 x 2.501 us = 89556.7 us
MULTI_ADD ADD: 17710 x 4.881 us = 86445.2 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 498.971 us = 192104 us (84.0548 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 246.581 us = 4.36697e+06 us (85.0406 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 305.086 us = 5.40309e+06 us (82.4606 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 293.79 us = 1.04061e+07 us (85.6381 GFLOPS/s)
MUL_MAT_VEC f16 m=1280 n=1 k=576 batch=20: 18048 x 583.603 us = 1.05329e+07 us (50.4891 GFLOPS/s)
MUL_MAT_VEC f16 m=256 n=1 k=576 batch=20: 47 x 70.986 us = 3336.36 us (83.0176 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=1280 batch=20: 18048 x 462.957 us = 8.35546e+06 us (56.6016 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=256 batch=20: 47 x 64.679 us = 3039.92 us (80.9015 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.354 us = 254220 us (18.2576 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 491.32 us = 378316 us (85.3472 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 71.58 us = 2.53539e+06 us (87.8717 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7207.37 us = 2.77484e+06 us (87.998 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.532 us = 1.196e+06 us (93.1319 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 247.305 us = 95212.8 us (84.7916 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 63.152 us = 1.14275e+06 us (82.9379 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 67.217 us = 1.21631e+06 us (58.3463 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 92.991 us = 1.68268e+06 us (84.5153 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.402 us = 332985 us (128.177 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.424 us = 876245 us (64.9454 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.147 us = 188282 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.823 us = 14897 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.478 us = 99126.1 us
ROPE: 36190 x 1.986 us = 71897.9 us
SET_ROWS: 36190 x 2.803 us = 101441 us
SOFT_MAX: 18095 x 10.584 us = 191526 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.366 us = 77327.7 us
Total time: 5.28724e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	pp512 @ d2048	73.51 ± 0.13

---

Vulkan Timings

ADD: 381 x 303.989 us = 115820 us
CONCAT: 752 x 603.49 us = 453825 us
CONT: 376 x 506.122 us = 190302 us
GET_ROWS: 10 x 2.796 us = 27.96 us
GLU: 738 x 566.85 us = 418336 us
MUL: 360 x 1195.58 us = 430410 us
MULTI_ADD ADD: 365 x 671.498 us = 245097 us
MUL_MAT f16 m=1024 n=512 k=576 batch=20: 47 x 11795.3 us = 554381 us (1023.21 GFLOPS/s)
MUL_MAT f16 m=1536 n=512 k=576 batch=20: 47 x 17452.5 us = 820268 us (1037.31 GFLOPS/s)
MUL_MAT f16 m=2048 n=512 k=576 batch=20: 47 x 22875.2 us = 1.07513e+06 us (1055.21 GFLOPS/s)
MUL_MAT f16 m=2560 n=512 k=576 batch=20: 141 x 28352.6 us = 3.99772e+06 us (1064.2 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=1024 batch=20: 47 x 9040.88 us = 424921 us (1187.07 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=1536 batch=20: 47 x 12991.6 us = 610603 us (1239.33 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=2048 batch=20: 47 x 17603.6 us = 827368 us (1219.62 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=2560 batch=20: 141 x 21172.2 us = 2.98528e+06 us (1267.62 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=512 batch=20: 94 x 4420.89 us = 415563 us (1213.21 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=576 batch=20: 94 x 6404.29 us = 602004 us (942.267 GFLOPS/s)
MUL_MAT f32 m=64 n=512 k=2048: 360 x 403.922 us = 145412 us (332.205 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 16 x 15066.2 us = 241060 us (1425.01 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 720 x 2279.28 us = 1.64108e+06 us (1412.92 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 8 x 15703.2 us = 125626 us (1367.48 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 360 x 2311.16 us = 832018 us (1393.32 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 376 x 7491.99 us = 2.81699e+06 us (1433.05 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 376 x 1896.94 us = 713249 us (1413.72 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 376 x 2195.64 us = 825560 us (914.55 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 376 x 2919.41 us = 1.0977e+06 us (1378.33 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 376 x 886.973 us = 333502 us (1361.56 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 376 x 1203.64 us = 452569 us (1337.79 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 720 x 21146.3 us = 1.52253e+07 us (609.173 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 360 x 21361.7 us = 7.69022e+06 us (602.981 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 5 x 350.6 us = 1753 us (71.7559 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 10 x 294.136 us = 2941.36 us (85.5376 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 5 x 14.352 us = 71.76 us (18.2609 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 10 x 73.58 us = 735.8 us (85.4841 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 5 x 7192.45 us = 35962.2 us (88.1805 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 5 x 74.888 us = 374.44 us (83.9842 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 10 x 3.576 us = 35.76 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 744 x 212.036 us = 157755 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 376 x 64.873 us = 24392.4 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 376 x 80.468 us = 30256 us
ROPE: 752 x 108.646 us = 81701.8 us
SET_ROWS: 752 x 123.704 us = 93026.1 us
SOFT_MAX: 376 x 5005.68 us = 1.88214e+06 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 365 x 15.302 us = 5585.28 us
Total time: 4.86241e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	tg128 @ d2048	5.13 ± 0.01

---

Vulkan Timings

ADD: 385 x 3.906 us = 1504.04 us
CONCAT: 36190 x 3.915 us = 141698 us
CONT: 18095 x 3.186 us = 57661.5 us
GET_ROWS: 770 x 3.654 us = 2814 us
GLU: 35805 x 2.425 us = 86860.2 us
MULTI_ADD ADD: 17710 x 4.815 us = 85278.7 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 489.486 us = 188452 us (85.6836 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 248.552 us = 4.40186e+06 us (84.3664 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 306.956 us = 5.4362e+06 us (81.9583 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 296.55 us = 1.05038e+07 us (84.8411 GFLOPS/s)
MUL_MAT_VEC f16 m=2304 n=1 k=576 batch=20: 18048 x 1142.53 us = 2.06204e+07 us (46.4217 GFLOPS/s)
MUL_MAT_VEC f16 m=256 n=1 k=576 batch=20: 47 x 62.323 us = 2929.2 us (94.5571 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=2304 batch=20: 18048 x 1047.36 us = 1.89028e+07 us (45.0423 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=256 batch=20: 47 x 64.742 us = 3042.88 us (80.8228 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.371 us = 254523 us (18.2358 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 490.117 us = 377390 us (85.5566 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 72.245 us = 2.55894e+06 us (87.0631 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7265.67 us = 2.79728e+06 us (87.2918 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.935 us = 1.20314e+06 us (92.5793 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 249.782 us = 96166.2 us (83.9509 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 63.344 us = 1.14621e+06 us (82.6873 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 67.633 us = 1.22383e+06 us (57.9879 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 93.713 us = 1.69574e+06 us (83.8645 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.508 us = 334918 us (127.437 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.771 us = 882514 us (64.484 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.2 us = 190200 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.775 us = 14030.2 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.53 us = 100070 us
ROPE: 36190 x 1.985 us = 71840.8 us
SET_ROWS: 36190 x 2.965 us = 107332 us
SOFT_MAX: 18095 x 14.636 us = 264854 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.427 us = 78412.1 us
Total time: 7.38327e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	pp512 @ d8096	38.62 ± 0.07

---

Vulkan Timings

ADD: 948 x 309.818 us = 293708 us
CONCAT: 1880 x 609.433 us = 1.14573e+06 us
CONT: 940 x 513.66 us = 482841 us
GET_ROWS: 16 x 2.79 us = 44.64 us
GLU: 1836 x 572.621 us = 1.05133e+06 us
MUL: 900 x 1184.42 us = 1.06598e+06 us
MULTI_ADD ADD: 908 x 681.852 us = 619122 us
MUL_MAT f16 m=1024 n=512 k=576 batch=20: 47 x 11845.6 us = 556741 us (1018.87 GFLOPS/s)
MUL_MAT f16 m=1536 n=512 k=576 batch=20: 47 x 17522.8 us = 823569 us (1033.15 GFLOPS/s)
MUL_MAT f16 m=2048 n=512 k=576 batch=20: 47 x 22914.4 us = 1.07697e+06 us (1053.41 GFLOPS/s)
MUL_MAT f16 m=2560 n=512 k=576 batch=20: 47 x 28660.3 us = 1.34703e+06 us (1052.77 GFLOPS/s)
MUL_MAT f16 m=3072 n=512 k=576 batch=20: 47 x 33899 us = 1.59325e+06 us (1068.09 GFLOPS/s)
MUL_MAT f16 m=3584 n=512 k=576 batch=20: 47 x 39278.6 us = 1.84609e+06 us (1075.44 GFLOPS/s)
MUL_MAT f16 m=4096 n=512 k=576 batch=20: 47 x 44658.2 us = 2.09893e+06 us (1081.02 GFLOPS/s)
MUL_MAT f16 m=4608 n=512 k=576 batch=20: 47 x 50262.2 us = 2.36232e+06 us (1080.55 GFLOPS/s)
MUL_MAT f16 m=512 n=416 k=8192 batch=20: 47 x 60810.7 us = 2.8581e+06 us (1147.64 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=1024 batch=20: 47 x 9132.06 us = 429207 us (1175.22 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=1536 batch=20: 47 x 13074.7 us = 614513 us (1231.45 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=2048 batch=20: 47 x 17841.5 us = 838551 us (1203.35 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=2560 batch=20: 47 x 21487.3 us = 1.0099e+06 us (1249.03 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=3072 batch=20: 47 x 25809.2 us = 1.21303e+06 us (1247.89 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=3584 batch=20: 47 x 29671.5 us = 1.39456e+06 us (1266.39 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=4096 batch=20: 47 x 34016 us = 1.59875e+06 us (1262.48 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=4608 batch=20: 47 x 37742 us = 1.77388e+06 us (1280.09 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=512 batch=20: 94 x 4474.95 us = 420645 us (1198.55 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=5120 batch=20: 47 x 42125.9 us = 1.97992e+06 us (1274.32 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=5632 batch=20: 47 x 46204.9 us = 2.17163e+06 us (1278.01 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=576 batch=20: 94 x 6487.67 us = 609841 us (930.157 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=6144 batch=20: 47 x 50565.8 us = 2.37659e+06 us (1273.97 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=6656 batch=20: 47 x 54260.4 us = 2.55024e+06 us (1286.17 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=7168 batch=20: 47 x 58874.7 us = 2.76711e+06 us (1276.55 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=7680 batch=20: 47 x 63288.7 us = 2.97457e+06 us (1272.35 GFLOPS/s)
MUL_MAT f16 m=512 n=512 k=8704 batch=20: 141 x 70478.6 us = 9.93749e+06 us (1294.9 GFLOPS/s)
MUL_MAT f16 m=5120 n=512 k=576 batch=20: 47 x 55621 us = 2.61419e+06 us (1084.94 GFLOPS/s)
MUL_MAT f16 m=5632 n=512 k=576 batch=20: 47 x 61123.3 us = 2.8728e+06 us (1086 GFLOPS/s)
MUL_MAT f16 m=6144 n=512 k=576 batch=20: 47 x 66553.5 us = 3.12801e+06 us (1088.07 GFLOPS/s)
MUL_MAT f16 m=6656 n=512 k=576 batch=20: 47 x 72104.7 us = 3.38892e+06 us (1087.99 GFLOPS/s)
MUL_MAT f16 m=7168 n=512 k=576 batch=20: 47 x 77421.3 us = 3.6388e+06 us (1091.22 GFLOPS/s)
MUL_MAT f16 m=7680 n=512 k=576 batch=20: 47 x 82702.5 us = 3.88702e+06 us (1094.51 GFLOPS/s)
MUL_MAT f16 m=8192 n=416 k=576 batch=20: 47 x 77162.8 us = 3.62665e+06 us (1016.67 GFLOPS/s)
MUL_MAT f16 m=8704 n=512 k=576 batch=20: 141 x 93564 us = 1.31925e+07 us (1096.44 GFLOPS/s)
MUL_MAT f32 m=64 n=416 k=2048: 45 x 349.983 us = 15749.2 us (311.516 GFLOPS/s)
MUL_MAT f32 m=64 n=512 k=2048: 855 x 395.237 us = 337928 us (339.505 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=416 k=2048: 2 x 12917.1 us = 25834.2 us (1350.46 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 38 x 15526.5 us = 590008 us (1382.77 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=416 k=2048: 90 x 1970.35 us = 177331 us (1327.99 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 1710 x 2285.3 us = 3.90786e+06 us (1409.2 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=416 k=10240: 1 x 13581.8 us = 13581.8 us (1284.62 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=416 k=1536: 45 x 2051.46 us = 92315.5 us (1275.38 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=416 k=5120: 47 x 6334.98 us = 297744 us (1377 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 19 x 15762 us = 299477 us (1362.38 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 855 x 2314.68 us = 1.97905e+06 us (1391.2 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 893 x 7371.35 us = 6.58262e+06 us (1456.5 GFLOPS/s)
MUL_MAT q8_0 m=256 n=416 k=512 batch=20: 47 x 1622.87 us = 76274.8 us (1342.63 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 893 x 1901.52 us = 1.69805e+06 us (1410.31 GFLOPS/s)
MUL_MAT q8_0 m=512 n=416 k=192 batch=20: 47 x 1901.47 us = 89369.1 us (858.03 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 893 x 2198.84 us = 1.96356e+06 us (913.219 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=416 k=768: 47 x 2556.78 us = 120169 us (1278.73 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 893 x 2936.05 us = 2.6219e+06 us (1370.52 GFLOPS/s)
MUL_MAT q8_0 m=576 n=416 k=2048: 47 x 753.308 us = 35405.5 us (1302.56 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 893 x 892.885 us = 797346 us (1352.54 GFLOPS/s)
MUL_MAT q8_0 m=768 n=416 k=2048: 47 x 1089.13 us = 51189.4 us (1201.23 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 893 x 1207.38 us = 1.07819e+06 us (1333.65 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=416: 90 x 19280.1 us = 1.73521e+06 us (542.862 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 1710 x 21040.8 us = 3.59799e+07 us (612.226 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=416: 45 x 19750.7 us = 888783 us (529.883 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 855 x 21355.9 us = 1.82593e+07 us (603.146 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 8 x 328.35 us = 2626.8 us (76.6183 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 16 x 298.062 us = 4769 us (84.4108 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 8 x 14.69 us = 117.52 us (17.8407 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 16 x 76.515 us = 1224.24 us (82.2051 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 8 x 7346.78 us = 58774.3 us (86.328 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 8 x 73.08 us = 584.64 us (86.062 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 16 x 3.652 us = 58.44 us
RMS_NORM_MUL RMS_NORM(2048,416,1,1): 93 x 173.96 us = 16178.3 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 1767 x 213.174 us = 376680 us
RMS_NORM_MUL RMS_NORM(512,416,1,1): 47 x 53.934 us = 2534.92 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 893 x 65.602 us = 58583.3 us
RMS_NORM_MUL RMS_NORM(768,416,1,1): 47 x 65.517 us = 3079.32 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 893 x 81.974 us = 73202.9 us
ROPE: 1880 x 108.902 us = 204736 us
SET_ROWS: 1880 x 124.398 us = 233870 us
SOFT_MAX: 940 x 14322.8 us = 1.34635e+07 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 908 x 15.09 us = 13701.8 us
Total time: 1.84459e+08 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	0	1	tg128 @ d8096	2.17 ± 0.02

---

Vulkan Timings

ADD: 385 x 4.081 us = 1571.52 us
CONCAT: 36190 x 3.972 us = 143779 us
CONT: 18095 x 3.202 us = 57956.2 us
GET_ROWS: 770 x 3.659 us = 2818.12 us
GLU: 35805 x 2.586 us = 92627.2 us
MULTI_ADD ADD: 17710 x 4.989 us = 88355.8 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 502.488 us = 193458 us (83.4666 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 248.648 us = 4.40356e+06 us (84.3338 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 308.206 us = 5.45833e+06 us (81.626 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 296.887 us = 1.05158e+07 us (84.7449 GFLOPS/s)
MUL_MAT_VEC f16 m=256 n=1 k=576 batch=20: 47 x 62.507 us = 2937.84 us (94.279 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=256 batch=20: 47 x 67.117 us = 3154.52 us (77.9624 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=8192 batch=20: 13536 x 3700.19 us = 5.00858e+07 us (45.3387 GFLOPS/s)
MUL_MAT_VEC f16 m=512 n=1 k=8448 batch=20: 4512 x 3811.31 us = 1.71966e+07 us (45.3924 GFLOPS/s)
MUL_MAT_VEC f16 m=8192 n=1 k=576 batch=20: 13536 x 4035.44 us = 5.46237e+07 us (46.7309 GFLOPS/s)
MUL_MAT_VEC f16 m=8448 n=1 k=576 batch=20: 4512 x 4151.79 us = 1.87329e+07 us (46.8408 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.444 us = 255817 us (18.1436 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 482.517 us = 371539 us (86.9041 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 72.362 us = 2.56308e+06 us (86.9224 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7263.97 us = 2.79663e+06 us (87.3123 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.965 us = 1.20366e+06 us (92.5386 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 252.51 us = 97216.4 us (83.0441 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 64.132 us = 1.16047e+06 us (81.6713 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 67.676 us = 1.2246e+06 us (57.9512 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 93.981 us = 1.70059e+06 us (83.625 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.475 us = 334309 us (127.67 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.838 us = 883728 us (64.3955 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.175 us = 189302 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.74 us = 13407.2 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.562 us = 100652 us
ROPE: 36190 x 2.003 us = 72523.1 us
SET_ROWS: 36190 x 3.701 us = 133974 us
SOFT_MAX: 18095 x 36.456 us = 659687 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.411 us = 78120 us
Total time: 1.75443e+08 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	pp512	85.93 ± 0.66

---

Vulkan Timings

ADD: 192 x 301.698 us = 57926 us
CONCAT: 376 x 610.258 us = 229457 us
CONT: 188 x 508.511 us = 95600.2 us
CPY: 4 x 47.23 us = 188.92 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,512,1,1),  v(512,512,1,1),  m(512,512,1,1): 188 x 33936.6 us = 6.38008e+06 us
GET_ROWS: 8 x 5.54 us = 44.32 us
GLU: 372 x 569.212 us = 211747 us
MUL: 180 x 1188.51 us = 213933 us
MULTI_ADD ADD: 184 x 667.776 us = 122871 us
MUL_MAT f32 m=64 n=512 k=2048: 180 x 382.687 us = 68883.8 us (350.638 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 8 x 14850.6 us = 118805 us (1445.71 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 360 x 2264.99 us = 815398 us (1421.83 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 4 x 15402.4 us = 61609.4 us (1394.19 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 180 x 2304.03 us = 414726 us (1397.63 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 188 x 7196.71 us = 1.35298e+06 us (1491.84 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 188 x 3085.74 us = 580119 us (869.073 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 188 x 2222.43 us = 417816 us (903.527 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 188 x 2910.61 us = 547194 us (1382.5 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 188 x 880.072 us = 165454 us (1372.23 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 188 x 1197.16 us = 225065 us (1345.04 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 360 x 21095.5 us = 7.59439e+06 us (610.639 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 180 x 21619.5 us = 3.89151e+06 us (595.791 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 4 x 323.19 us = 1292.76 us (77.8416 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 8 x 300.27 us = 2402.16 us (83.7902 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 4 x 14.77 us = 59.08 us (17.7441 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 8 x 74.68 us = 597.44 us (84.225 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 4 x 7145.03 us = 28580.1 us (88.7657 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 4 x 67.11 us = 268.44 us (93.7179 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 8 x 3.53 us = 28.24 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 372 x 209.82 us = 78053.2 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 188 x 65.395 us = 12294.4 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 188 x 81.057 us = 15238.8 us
ROPE: 376 x 109.615 us = 41215.4 us
SET_ROWS: 376 x 68.79 us = 25865.2 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 184 x 14.968 us = 2754.12 us
Total time: 2.37744e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	tg128	10.67 ± 0.00

---

Vulkan Timings

ADD: 385 x 4.005 us = 1542.08 us
CONCAT: 36190 x 4.542 us = 164408 us
CONT: 18095 x 3.187 us = 57676.8 us
CPY: 385 x 1.568 us = 603.91 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,256,1,1),  v(512,256,1,1),  m(256,1,1,1): 18095 x 72.255 us = 1.30746e+06 us
GET_ROWS: 770 x 1.594 us = 1227.76 us
GLU: 35805 x 2.425 us = 86860.5 us
MULTI_ADD ADD: 17710 x 4.801 us = 85027 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 497.591 us = 191573 us (84.288 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 246.119 us = 4.35877e+06 us (85.2005 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 302.757 us = 5.36184e+06 us (83.0949 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 292.811 us = 1.03714e+07 us (85.9245 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.288 us = 253055 us (18.3416 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 490.214 us = 377465 us (85.5397 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 71.192 us = 2.52165e+06 us (88.3505 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7162.8 us = 2.75768e+06 us (88.5455 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.064 us = 1.18771e+06 us (93.7813 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 245.715 us = 94600.3 us (85.3406 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 67.417 us = 1.21992e+06 us (77.6911 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 67.904 us = 1.22873e+06 us (57.7565 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 92.385 us = 1.67172e+06 us (85.0694 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.325 us = 331601 us (128.712 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.054 us = 869549 us (65.4455 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.146 us = 188223 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.723 us = 13093.6 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.525 us = 99982 us
ROPE: 36190 x 0.896 us = 32454 us
SET_ROWS: 36190 x 2.032 us = 73563 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.348 us = 77019.2 us
Total time: 3.49864e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	pp512 @ d1024	50.36 ± 0.06

---

Vulkan Timings

ADD: 287 x 302.427 us = 86796.8 us
CONCAT: 564 x 608.659 us = 343284 us
CONT: 282 x 507.995 us = 143255 us
CPY: 6 x 89.84 us = 539.04 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,1024,1,1),  v(512,1024,1,1),  m(1024,512,1,1): 47 x 79928.9 us = 3.75666e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,1536,1,1),  v(512,1536,1,1),  m(1536,512,1,1): 141 x 124821 us = 1.75998e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,512,1,1),  v(512,512,1,1),  m(512,512,1,1): 94 x 34008.1 us = 3.19676e+06 us
GET_ROWS: 10 x 4.944 us = 49.44 us
GLU: 556 x 572.061 us = 318066 us
MUL: 270 x 1188.46 us = 320884 us
MULTI_ADD ADD: 275 x 672.472 us = 184930 us
MUL_MAT f32 m=64 n=512 k=2048: 270 x 386.873 us = 104456 us (346.845 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 12 x 14791 us = 177493 us (1451.53 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 540 x 2264.72 us = 1.22295e+06 us (1422 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 6 x 15377.7 us = 92266 us (1396.43 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 270 x 2308.53 us = 623303 us (1394.9 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 282 x 7206.06 us = 2.03211e+06 us (1489.91 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 282 x 3085.44 us = 870094 us (869.158 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 282 x 2182.11 us = 615355 us (920.222 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 282 x 2906.11 us = 819524 us (1384.64 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 282 x 883.497 us = 249146 us (1366.91 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 282 x 1196.73 us = 337478 us (1345.52 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 540 x 20853.1 us = 1.12607e+07 us (617.739 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 270 x 21410.2 us = 5.78075e+06 us (601.616 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 5 x 334.544 us = 1672.72 us (75.1998 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 10 x 297.6 us = 2976 us (84.5419 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 5 x 14.128 us = 70.64 us (18.5504 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 10 x 77.132 us = 771.32 us (81.5475 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 5 x 7165.7 us = 35828.5 us (88.5097 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 5 x 70.752 us = 353.76 us (88.8937 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 10 x 3.488 us = 34.88 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 558 x 209.664 us = 116993 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 282 x 65.346 us = 18427.7 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 282 x 80.988 us = 22838.8 us
ROPE: 564 x 107.288 us = 60510.8 us
SET_ROWS: 564 x 66.962 us = 37766.6 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 275 x 15.083 us = 4147.88 us
Total time: 5.0439e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	tg128 @ d1024	9.56 ± 0.01

---

Vulkan Timings

ADD: 385 x 4.163 us = 1603.08 us
CONCAT: 36190 x 4.613 us = 166968 us
CONT: 18095 x 3.203 us = 57975.5 us
CPY: 385 x 1.431 us = 551.2 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,1280,1,1),  v(512,1280,1,1),  m(1280,1,1,1): 18048 x 304.271 us = 5.49149e+06 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,256,1,1),  v(512,256,1,1),  m(256,1,1,1): 47 x 60.448 us = 2841.08 us
GET_ROWS: 770 x 1.646 us = 1268.12 us
GLU: 35805 x 2.474 us = 88616.1 us
MULTI_ADD ADD: 17710 x 4.98 us = 88208.7 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 497.363 us = 191485 us (84.3266 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 245.289 us = 4.34408e+06 us (85.4886 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 303.561 us = 5.37607e+06 us (82.8749 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 292.605 us = 1.03641e+07 us (85.985 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.213 us = 251714 us (18.4393 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 489.456 us = 376881 us (85.6722 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 71.219 us = 2.52258e+06 us (88.318 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7165.29 us = 2.75864e+06 us (88.5147 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.044 us = 1.18735e+06 us (93.8099 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 244.361 us = 94079.3 us (85.8132 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 66.53 us = 1.20387e+06 us (78.7269 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 67.2 us = 1.21599e+06 us (58.3615 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 92.485 us = 1.67353e+06 us (84.9773 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.255 us = 330342 us (129.203 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.168 us = 871600 us (65.2915 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.2 us = 190194 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.669 us = 12120.2 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.579 us = 100965 us
ROPE: 36190 x 1.985 us = 71840.1 us
SET_ROWS: 36190 x 1.368 us = 49537.9 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.37 us = 77392.7 us
Total time: 3.91639e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	pp512 @ d2048	35.55 ± 0.02

---

Vulkan Timings

ADD: 381 x 306.013 us = 116591 us
CONCAT: 752 x 604.587 us = 454650 us
CONT: 376 x 504.093 us = 189539 us
CPY: 8 x 126.94 us = 1015.52 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,1024,1,1),  v(512,1024,1,1),  m(1024,512,1,1): 47 x 80132.1 us = 3.76621e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,1536,1,1),  v(512,1536,1,1),  m(1536,512,1,1): 47 x 125161 us = 5.88254e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,2048,1,1),  v(512,2048,1,1),  m(2048,512,1,1): 47 x 170687 us = 8.02227e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,2560,1,1),  v(512,2560,1,1),  m(2560,512,1,1): 141 x 215381 us = 3.03687e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,512,1,1),  v(512,512,1,1),  m(512,512,1,1): 94 x 33979.5 us = 3.19407e+06 us
GET_ROWS: 10 x 5.812 us = 58.12 us
GLU: 738 x 567.979 us = 419169 us
MUL: 360 x 1185.31 us = 426711 us
MULTI_ADD ADD: 365 x 666.553 us = 243292 us
MUL_MAT f32 m=64 n=512 k=2048: 360 x 386.565 us = 139164 us (347.121 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 16 x 14749 us = 235984 us (1455.66 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 720 x 2255.99 us = 1.62431e+06 us (1427.51 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 8 x 15319.2 us = 122554 us (1401.75 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 360 x 2312.53 us = 832512 us (1392.49 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 376 x 7191.06 us = 2.70384e+06 us (1493.02 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 376 x 3089.29 us = 1.16157e+06 us (868.073 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 376 x 2188.81 us = 822993 us (917.403 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 376 x 2902.96 us = 1.09151e+06 us (1386.14 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 376 x 880.078 us = 330910 us (1372.22 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 376 x 1196.94 us = 450050 us (1345.28 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 720 x 20482.7 us = 1.47476e+07 us (628.909 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 360 x 21039.6 us = 7.57427e+06 us (612.211 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 5 x 337.184 us = 1685.92 us (74.611 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 10 x 288.18 us = 2881.8 us (87.3054 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 5 x 14.192 us = 70.96 us (18.4667 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 10 x 72.636 us = 726.36 us (86.5951 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 5 x 7146.48 us = 35732.4 us (88.7477 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 5 x 66.904 us = 334.52 us (94.0065 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 10 x 3.544 us = 35.44 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 744 x 207.738 us = 154558 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 376 x 64.713 us = 24332.4 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 376 x 80.233 us = 30167.7 us
ROPE: 752 x 108.248 us = 81402.6 us
SET_ROWS: 752 x 68.544 us = 51545.4 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 365 x 14.832 us = 5413.72 us
Total time: 8.53109e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	tg128 @ d2048	8.61 ± 0.00

---

Vulkan Timings

ADD: 385 x 4.035 us = 1553.84 us
CONCAT: 36190 x 4.672 us = 169114 us
CONT: 18095 x 3.17 us = 57362.3 us
CPY: 385 x 1.156 us = 445.28 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,2304,1,1),  v(512,2304,1,1),  m(2304,1,1,1): 18048 x 549.62 us = 9.91956e+06 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,256,1,1),  v(512,256,1,1),  m(256,1,1,1): 47 x 63.462 us = 2982.72 us
GET_ROWS: 770 x 1.607 us = 1237.4 us
GLU: 35805 x 2.401 us = 85978.3 us
MULTI_ADD ADD: 17710 x 4.898 us = 86754.4 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 496.889 us = 191302 us (84.4072 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 245.743 us = 4.35212e+06 us (85.3308 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 303.03 us = 5.36667e+06 us (83.0202 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 293.411 us = 1.03926e+07 us (85.7488 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.356 us = 254255 us (18.255 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 490.397 us = 377606 us (85.5078 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 71.165 us = 2.52068e+06 us (88.3845 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7178.83 us = 2.76385e+06 us (88.3478 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 67.061 us = 1.18765e+06 us (93.7862 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 246.316 us = 94831.8 us (85.1322 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 66.938 us = 1.21124e+06 us (78.2478 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 67.284 us = 1.21752e+06 us (58.2883 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 92.49 us = 1.67361e+06 us (84.9734 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.351 us = 332074 us (128.529 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.196 us = 872116 us (65.2528 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.207 us = 190449 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.719 us = 13028.3 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.557 us = 100560 us
ROPE: 36190 x 2.016 us = 72967 us
SET_ROWS: 36190 x 1.36 us = 49251 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.332 us = 76724.8 us
Total time: 4.36361e+07 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	pp512 @ d8096	12.98 ± 0.00

---

Vulkan Timings

ADD: 948 x 304.117 us = 288303 us
CONCAT: 1880 x 578.476 us = 1.08754e+06 us
CONT: 940 x 497.973 us = 468095 us
CPY: 20 x 357.626 us = 7152.52 us
FLASH_ATTN_EXT dst(512,20,416,1),  q(576,416,20,1),  k(576,8192,1,1),  v(512,8192,1,1),  m(8192,416,1,1): 47 x 574728 us = 2.70122e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,1024,1,1),  v(512,1024,1,1),  m(1024,512,1,1): 47 x 80109.3 us = 3.76514e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,1536,1,1),  v(512,1536,1,1),  m(1536,512,1,1): 47 x 125185 us = 5.88369e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,2048,1,1),  v(512,2048,1,1),  m(2048,512,1,1): 47 x 170731 us = 8.02437e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,2560,1,1),  v(512,2560,1,1),  m(2560,512,1,1): 47 x 215291 us = 1.01187e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,3072,1,1),  v(512,3072,1,1),  m(3072,512,1,1): 47 x 260502 us = 1.22436e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,3584,1,1),  v(512,3584,1,1),  m(3584,512,1,1): 47 x 305306 us = 1.43494e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,4096,1,1),  v(512,4096,1,1),  m(4096,512,1,1): 47 x 350532 us = 1.6475e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,4608,1,1),  v(512,4608,1,1),  m(4608,512,1,1): 47 x 395336 us = 1.85808e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,512,1,1),  v(512,512,1,1),  m(512,512,1,1): 94 x 34191.5 us = 3.214e+06 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,5120,1,1),  v(512,5120,1,1),  m(5120,512,1,1): 47 x 440387 us = 2.06982e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,5632,1,1),  v(512,5632,1,1),  m(5632,512,1,1): 47 x 485244 us = 2.28065e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,6144,1,1),  v(512,6144,1,1),  m(6144,512,1,1): 47 x 531778 us = 2.49936e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,6656,1,1),  v(512,6656,1,1),  m(6656,512,1,1): 47 x 575986 us = 2.70713e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,7168,1,1),  v(512,7168,1,1),  m(7168,512,1,1): 47 x 621233 us = 2.91979e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,7680,1,1),  v(512,7680,1,1),  m(7680,512,1,1): 47 x 665887 us = 3.12967e+07 us
FLASH_ATTN_EXT dst(512,20,512,1),  q(576,512,20,1),  k(576,8704,1,1),  v(512,8704,1,1),  m(8704,512,1,1): 141 x 747361 us = 1.05378e+08 us
GET_ROWS: 16 x 2.962 us = 47.4 us
GLU: 1836 x 563.043 us = 1.03375e+06 us
MUL: 900 x 1176.42 us = 1.05878e+06 us
MULTI_ADD ADD: 908 x 661.79 us = 600906 us
MUL_MAT f32 m=64 n=416 k=2048: 45 x 345.337 us = 15540.2 us (315.706 GFLOPS/s)
MUL_MAT f32 m=64 n=512 k=2048: 855 x 387.95 us = 331698 us (345.881 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=416 k=2048: 2 x 12715.5 us = 25431.1 us (1371.87 GFLOPS/s)
MUL_MAT q8_0 m=10240 n=512 k=2048: 38 x 14971.7 us = 568926 us (1434.01 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=416 k=2048: 90 x 1949.3 us = 175437 us (1342.33 GFLOPS/s)
MUL_MAT q8_0 m=1536 n=512 k=2048: 1710 x 2250.75 us = 3.84878e+06 us (1430.83 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=416 k=10240: 1 x 13437.4 us = 13437.4 us (1298.43 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=416 k=1536: 45 x 2004.3 us = 90193.5 us (1305.39 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=416 k=5120: 47 x 6291.02 us = 295678 us (1386.63 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=10240: 19 x 15498.8 us = 294478 us (1385.51 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=1536: 855 x 2276.73 us = 1.94661e+06 us (1414.39 GFLOPS/s)
MUL_MAT q8_0 m=2048 n=512 k=5120: 893 x 7193.46 us = 6.42376e+06 us (1492.52 GFLOPS/s)
MUL_MAT q8_0 m=256 n=416 k=512 batch=20: 47 x 2669.12 us = 125449 us (816.34 GFLOPS/s)
MUL_MAT q8_0 m=256 n=512 k=512 batch=20: 893 x 3081.57 us = 2.75184e+06 us (870.249 GFLOPS/s)
MUL_MAT q8_0 m=512 n=416 k=192 batch=20: 47 x 1886.83 us = 88681.1 us (864.687 GFLOPS/s)
MUL_MAT q8_0 m=512 n=512 k=192 batch=20: 893 x 2176.05 us = 1.94321e+06 us (922.783 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=416 k=768: 47 x 2547.47 us = 119731 us (1283.4 GFLOPS/s)
MUL_MAT q8_0 m=5120 n=512 k=768: 893 x 2906.27 us = 2.5953e+06 us (1384.56 GFLOPS/s)
MUL_MAT q8_0 m=576 n=416 k=2048: 47 x 754.071 us = 35441.4 us (1301.24 GFLOPS/s)
MUL_MAT q8_0 m=576 n=512 k=2048: 893 x 879.234 us = 785156 us (1373.54 GFLOPS/s)
MUL_MAT q8_0 m=768 n=416 k=2048: 47 x 1083.13 us = 50907.3 us (1207.89 GFLOPS/s)
MUL_MAT q8_0 m=768 n=512 k=2048: 893 x 1193.89 us = 1.06615e+06 us (1348.71 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=416: 90 x 18503.7 us = 1.66534e+06 us (565.638 GFLOPS/s)
MUL_MAT_ID q8_0 m=1536 n=4 k=2048 n_expert=64 batch=512: 1710 x 20453.3 us = 3.49751e+07 us (629.813 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=416: 45 x 19095.7 us = 859307 us (548.059 GFLOPS/s)
MUL_MAT_ID q8_0 m=2048 n=4 k=1536 n_expert=64 batch=512: 855 x 21036 us = 1.79858e+07 us (612.318 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 8 x 338.915 us = 2711.32 us (74.2299 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 16 x 293.117 us = 4689.88 us (85.8348 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 8 x 14.34 us = 114.72 us (18.2762 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 16 x 74.732 us = 1195.72 us (84.1658 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 8 x 7134.88 us = 57079 us (88.892 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 8 x 68.99 us = 551.92 us (91.1641 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 16 x 3.537 us = 56.6 us
RMS_NORM_MUL RMS_NORM(2048,416,1,1): 93 x 169.918 us = 15802.4 us
RMS_NORM_MUL RMS_NORM(2048,512,1,1): 1767 x 207.323 us = 366341 us
RMS_NORM_MUL RMS_NORM(512,416,1,1): 47 x 53.511 us = 2515.04 us
RMS_NORM_MUL RMS_NORM(512,512,1,1): 893 x 64.665 us = 57746.2 us
RMS_NORM_MUL RMS_NORM(768,416,1,1): 47 x 64.828 us = 3046.96 us
RMS_NORM_MUL RMS_NORM(768,512,1,1): 893 x 80.025 us = 71463 us
ROPE: 1880 x 99.024 us = 186166 us
SET_ROWS: 1880 x 67.692 us = 127261 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 908 x 14.789 us = 13428.7 us
Total time: 4.65641e+08 us.

model	size	params	backend	threads	fa	dio	test	t/s
deepseek2 ?B Q8_0	29.65 GiB	29.94 B	Vulkan,BLAS	8	1	1	tg128 @ d8096	5.49 ± 0.00

---

Vulkan Timings

ADD: 385 x 4.165 us = 1603.76 us
CONCAT: 36190 x 2.965 us = 107316 us
CONT: 18095 x 3.158 us = 57151.4 us
CPY: 385 x 0.871 us = 335.64 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,256,1,1),  v(512,256,1,1),  m(256,1,1,1): 47 x 63.187 us = 2969.8 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,8192,1,1),  v(512,8192,1,1),  m(8192,1,1,1): 13536 x 1947.55 us = 2.6362e+07 us
FLASH_ATTN_EXT dst(512,20,1,1),  q(576,1,20,1),  k(576,8448,1,1),  v(512,8448,1,1),  m(8448,1,1,1): 4512 x 1970.92 us = 8.89281e+06 us
GET_ROWS: 770 x 2.901 us = 2234.36 us
GLU: 35805 x 2.391 us = 85637.6 us
MULTI_ADD ADD: 17710 x 4.84 us = 85719 us
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=10240: 385 x 482.027 us = 185580 us (87.0096 GFLOPS/s)
MUL_MAT_ADD MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 17710 x 244.987 us = 4.33872e+06 us (85.5941 GFLOPS/s)
MUL_MAT_ID_MUL MUL_MAT_ID_VEC q8_0 m=2048 n=4 k=1536 n_expert=64: 17710 x 303.322 us = 5.37184e+06 us (82.9402 GFLOPS/s)
MUL_MAT_ID_VEC q8_0 m=1536 n=4 k=2048 n_expert=64: 35420 x 292.384 us = 1.03562e+07 us (86.0501 GFLOPS/s)
MUL_MAT_VEC f32 m=64 n=1 k=2048: 17710 x 14.316 us = 253542 us (18.3064 GFLOPS/s)
MUL_MAT_VEC q8_0 m=10240 n=1 k=2048: 770 x 484.219 us = 372849 us (86.5988 GFLOPS/s)
MUL_MAT_VEC q8_0 m=1536 n=1 k=2048: 35420 x 71.1 us = 2.51838e+06 us (88.4651 GFLOPS/s)
MUL_MAT_VEC q8_0 m=154880 n=1 k=2048: 385 x 7148.5 us = 2.75217e+06 us (88.7226 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=1536: 17710 x 66.93 us = 1.18534e+06 us (93.969 GFLOPS/s)
MUL_MAT_VEC q8_0 m=2048 n=1 k=5120: 385 x 245.295 us = 94438.9 us (85.4865 GFLOPS/s)
MUL_MAT_VEC q8_0 m=256 n=1 k=512 batch=20: 18095 x 66.214 us = 1.19815e+06 us (79.103 GFLOPS/s)
MUL_MAT_VEC q8_0 m=512 n=1 k=192 batch=20: 18095 x 69.489 us = 1.25741e+06 us (56.4392 GFLOPS/s)
MUL_MAT_VEC q8_0 m=5120 n=1 k=768: 18095 x 92.403 us = 1.67205e+06 us (85.0528 GFLOPS/s)
MUL_MAT_VEC q8_0 m=576 n=1 k=2048: 18095 x 18.224 us = 329773 us (129.426 GFLOPS/s)
MUL_MAT_VEC q8_0 m=768 n=1 k=2048: 18095 x 48.102 us = 870406 us (65.3811 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(2048,1,1,1): 36575 x 5.176 us = 189337 us
RMS_NORM_MUL RMS_NORM(512,1,1,1): 18095 x 0.804 us = 14549.9 us
RMS_NORM_MUL RMS_NORM(768,1,1,1): 18095 x 5.488 us = 99320.4 us
ROPE: 36190 x 1.188 us = 43018.2 us
SET_ROWS: 36190 x 1.386 us = 50184.9 us
TOPK_MOE_SIGMOID_NORM_BIAS SIGMOID: 17710 x 4.367 us = 77341.4 us
Total time: 6.88284e+07 us.

build: 8f91ca5 (7822)

[engrtipusultan]

@jeffbolznv is there environment variable to check what is going to CPU. In nvtop I am often seeing CPU going high especially when FA = on.

[0cc4m]

If you run llama-bench with --verbose it will show the usual model loading output. There you can see how much has gone to the GPU and how much has not. A typical sign for something falling back to CPU would be a graph splits number higher than the usual ~2 or so, for example.

[engrtipusultan]

@0cc4m thank you for information.

sched_reserve: graph splits = 2 does it mean it is always to some extent falling back to CPU?

Whenever you have free time, can you look at this at your AMD hardwares. Is this normal for this model or something needs to be improved going forward in llama.cpp.

Also there is, I though direct io was fixed. Is this the same on your devices.

load_all_data: using async uploads for device Vulkan0, buffer type Vulkan0, backend Vulkan0
read_raw_unsafe: Falling back to buffered IO due to Bad address
load_all_data: buffer type Vulkan_Host is not the default buffer type for device Vulkan0 for async uploads

verbos_llamabench_ouput.log

[rezidue]

I opened an issue yesterday (#19080) that I ran into while looking into performance issues of glm flash as context grew. I saw llama-bench crashing when running depth of 32k with rocm fa0 but it ended up not being model specific and related to OOM. What you might find interesting is the benchmarks I provided with fa0/1 for rocm and vulkan.

./build/bin/llama-bench -m /mnt/glycerin/models/unsloth_GLM-4.7-Flash-GGUF_GLM-4.7-Flash-Q4_K_M.gguf --device rocm0,vulkan1 -p 512,4096 -n 128,1024 -d 0,16384 -r 1 -fa 0,1
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx1036 (0x1036), VMM: no, Wave Size: 32
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl | fa | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------ | --------------: | -------------------: |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |           pp512 |        526.74 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |          pp4096 |        459.12 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |           tg128 |         70.08 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |          tg1024 |         67.93 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |  pp512 @ d16384 |        223.63 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        | pp4096 @ d16384 |        204.69 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |  tg128 @ d16384 |         16.73 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        | tg1024 @ d16384 |         16.42 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |           pp512 |        636.70 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |          pp4096 |        141.50 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |           tg128 |         56.33 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |          tg1024 |         44.52 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |  pp512 @ d16384 |         19.25 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        | pp4096 @ d16384 |         16.59 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |  tg128 @ d16384 |          6.33 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        | tg1024 @ d16384 |          6.24 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |           pp512 |       1020.43 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |          pp4096 |        883.52 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |           tg128 |         70.11 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |          tg1024 |         68.91 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |  pp512 @ d16384 |        258.00 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      | pp4096 @ d16384 |        254.41 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |  tg128 @ d16384 |         33.83 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      | tg1024 @ d16384 |         33.92 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |           pp512 |        915.04 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |          pp4096 |        616.07 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |           tg128 |         70.39 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |          tg1024 |         70.31 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |  pp512 @ d16384 |        142.27 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      | pp4096 @ d16384 |        130.63 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |  tg128 @ d16384 |         48.72 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      | tg1024 @ d16384 |         50.06 ± 0.00 |

[rezidue]

You might want to test build 7833. I was running 7801 and I've seen improvements everywhere.

./build/bin/llama-bench -m /mnt/glycerin/models/unsloth_GLM-4.7-Flash-GGUF_GLM-4.7-Flash-Q4_K_M.gguf --device rocm0,vulkan1 -p 512,4096 -n 128,1024 -d 0,16384 -r 1 -fa 0,1
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon AI PRO R9700, gfx1201 (0x1201), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx1036 (0x1036), VMM: no, Wave Size: 32
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl | fa | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------ | --------------: | -------------------: |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |           pp512 |        534.77 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |          pp4096 |        490.76 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |           tg128 |         67.17 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |          tg1024 |         64.06 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |  pp512 @ d16384 |        203.19 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        | pp4096 @ d16384 |        205.14 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        |  tg128 @ d16384 |         13.42 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | ROCm0        | tg1024 @ d16384 |         13.17 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |           pp512 |       2384.41 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |          pp4096 |       1419.45 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |           tg128 |         77.31 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |          tg1024 |         76.74 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |  pp512 @ d16384 |        326.18 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        | pp4096 @ d16384 |        300.36 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        |  tg128 @ d16384 |         57.03 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | ROCm0        | tg1024 @ d16384 |         57.39 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |           pp512 |       1927.26 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |          pp4096 |       1342.97 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |           tg128 |        112.56 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |          tg1024 |        111.24 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |  pp512 @ d16384 |        329.05 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      | pp4096 @ d16384 |        297.06 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      |  tg128 @ d16384 |         38.02 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  0 | Vulkan1      | tg1024 @ d16384 |         37.53 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |           pp512 |       1665.79 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |          pp4096 |        829.65 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |           tg128 |        114.82 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |          tg1024 |        114.55 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |  pp512 @ d16384 |        154.46 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      | pp4096 @ d16384 |        140.57 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      |  tg128 @ d16384 |         73.20 ± 0.00 |
| deepseek2 ?B Q4_K - Medium     |  17.05 GiB |    29.94 B | ROCm,Vulkan |  99 |  1 | Vulkan1      | tg1024 @ d16384 |         72.73 ± 0.00 |

[jdluzen]

I am also having oddities/slowness with AMD 7900 XTX using Vulkan. In regular conversation usage, it seems to work just fine: think then output, think then output.
Using Kilo code it appears to be doing something before thinking, then thinking, then output. Watching Task Manager, the Compute vs 3D graph behavior is also different. The something is very spikey in the Compute usage, and low in the 3D usage. While thinking and output both max out the Compute as expected, leaving 3D practically unused. The something takes forever, many times longer than thinking and output combined.
Using b7833 and llama-server.exe -m GLM-4.7-Flash-UD-Q4_K_XL.gguf --jinja --threads -1 --ctx-size 64000 --temp 0.7 --top-p 1.0 --min-p 0.01 --fit on

[ayylmaonade]

Even with the latest builds, on my RX 7900 XTX w/ Vulkan, the PP is still abysmal, especially once context reaches around ~10-15K tokens. Flash attention or not, it's far slower than comparable 30B-A3B models on my end. It's strange.

[jeffbolznv]

I think GLM-4.7 is fundamentally going to be slower than these other models for large context because the head size is so much larger (512/576, vs I think 128/128). You'll see the same thing with deepseek2. Here's the perf I see with different coopmat modes:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -m c:\models\GLM-4.7-Flash-Q4_K_M.gguf -p 512 -n 128 -d 0,4096,16384
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
load_backend: loaded Vulkan backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-vulkan.dll
load_backend: loaded CPU backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-cpu.dll
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |           pp512 |      8428.85 ± 72.15 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |           tg128 |        224.53 ± 3.87 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |   pp512 @ d4096 |      4555.33 ± 31.01 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |   tg128 @ d4096 |       176.54 ± 18.70 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |  pp512 @ d16384 |       1907.95 ± 4.71 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |  tg128 @ d16384 |        157.54 ± 0.89 |

build: 0440bfd16 (7835)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_DISABLE_COOPMAT2=1

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -m c:\models\GLM-4.7-Flash-Q4_K_M.gguf -p 512 -n 128 -d 0,4096,16384
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-vulkan.dll
load_backend: loaded CPU backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-cpu.dll
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |           pp512 |      4391.27 ± 33.43 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |           tg128 |       215.76 ± 17.75 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |   pp512 @ d4096 |        895.67 ± 0.81 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |   tg128 @ d4096 |        177.38 ± 5.24 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |  pp512 @ d16384 |        59.98 ± 76.13 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |  tg128 @ d16384 |        114.14 ± 0.53 |

build: 0440bfd16 (7835)

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>set GGML_VK_DISABLE_COOPMAT=1

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -m c:\models\GLM-4.7-Flash-Q4_K_M.gguf -p 512 -n 128 -d 0,4096,16384
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-vulkan.dll
load_backend: loaded CPU backend from Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo\ggml-cpu.dll
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |           pp512 |       2581.53 ± 5.94 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |           tg128 |       172.63 ± 37.41 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |   pp512 @ d4096 |        787.57 ± 0.79 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |   tg128 @ d4096 |        187.75 ± 1.15 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |  pp512 @ d16384 |        57.79 ± 71.79 |
| deepseek2 ?B Q4_K - Medium     |  16.88 GiB |    29.94 B | Vulkan     |  99 |  1 |  tg128 @ d16384 |        116.54 ± 0.26 |

The coopmat1/scalar paths clearly have room for improvement. There may be some tuning we can do with tile sizes to improve the prompt processing. For coopmat1 we should do that tuning after #19075 lands. For token gen, I think coopmat2 does better because it's able to use grouped query attention.

[jdluzen]

Using the new GLM-4.7-Flash-REAP-23B-A3B-UD-Q4_K_XL.gguf with --n-cpu-moe 2 or 4 work much better, though the initial pre-thinking lag is still there. It cuts the time down >75% or so on my single page HTML test prompt I've been using.

[jeffbolznv]

The coopmat1 path is falling back to the scalar path for FA due to needing too much shared memory. This is easier to fix with the changes in #19075, I'll leave a suggestion there.

[engrtipusultan]

@jeffbolznv GLM-4.7 Flash has MLA that was not there in GLM 4.7. Ideally that should reduce memery usage in bigger contexts.

#18986 where MLA (Multi-head Latent Attention) is implemented ggreg says

Change the concat layout of the MLA cache. The old was [pe, kv]. The new one is [kv, pe]. This has certain implications for backends such as Vulkan and overall it's better layout in terms of memory alignment

Todo
Try to take advantage of "V is view of K" in other backends

It was done for CUDA #19057 I am no expert will it be required separately for VULKAN?

As per ggreg in #19067 after this PR memery usage is reduced by 50%. So bottleneck of memory due to large head size should reduce.