0905129e9d
Apply suggestions from code review
...
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-26 18:03:58 +02:00
d8955f68e1
Ditch lane-parsing approach in favor of hard-coded patterns
2026-06-17 09:31:45 +02:00
89fcfc21d4
Remove gate_first from test_mul_mat_vec_fusion
2026-06-17 09:31:45 +02:00
8dd49e004f
Allow views to weights in ggml_can_fuse_subgraph
2026-06-17 09:31:45 +02:00
6b9024e842
Move gate_bias and scale into the same active-thread guard
...
Latest perf numbers:
B6000
build: 5b7d9f272 (9578)
+ ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite
| CPU | Model | Test | t/s master | t/s osimons/nvfp4_fuse_mmvq | Speedup |
|:----------------------------|:-------------------------|:-------------|-------------:|------------------------------:|----------:|
| INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B NVFP4 | tg128@d32768 | 151.79 | 154.10 | 1.02 |
| INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B Q4_K_M | tg128@d32768 | 187.90 | 187.27 | 1.00 |
| INTEL(R) XEON(R) GOLD 6542Y | gpt-oss 20B MXFP4 MoE | tg128@d32768 | 303.77 | 306.56 | 1.01 |
| INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B NVFP4 | tg128@d32768 | 193.41 | 207.99 | 1.08 |
| INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B Q4_K_M | tg128@d32768 | 217.60 | 218.58 | 1.00 |
DGX Spark
build: 5b7d9f272 (9578)
+ ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite
| CPU | Model | Test | t/s master | t/s osimons/nvfp4_fuse_mmvq | Speedup |
|:------|:-------------------------|:-------------|-------------:|------------------------------:|----------:|
| CPU | gemma4 26B.A4B NVFP4 | tg128@d32768 | 34.61 | 34.84 | 1.01 |
| CPU | gemma4 26B.A4B Q4_K_M | tg128@d32768 | 46.95 | 46.90 | 1.00 |
| CPU | gpt-oss 20B MXFP4 MoE | tg128@d32768 | 64.84 | 64.62 | 1.00 |
| CPU | qwen35moe 35B.A3B NVFP4 | tg128@d32768 | 59.63 | 60.72 | 1.02 |
| CPU | qwen35moe 35B.A3B Q4_K_M | tg128@d32768 | 56.53 | 56.55 | 1.00 |
PPL values for 5 chunks:
this PR
model mode ppl uncertainty log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_enabled 5.2892 0.35389 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_enabled.log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_disabled 5.2742 0.35215 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_disabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_enabled 5.4487 0.36866 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_disabled 5.4403 0.36782 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_disabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_enabled 17342.4348 3703.13932 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_disabled 18627.0624 3998.42475 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_disabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_enabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_enabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_disabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_disabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_enabled 17330.3926 3716.70472 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_enabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_disabled 17933.9524 3883.17066 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_disabled.log
master:
summary: ppl-value-checks/summary.tsv
model mode ppl uncertainty log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_enabled 5.2892 0.35389 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_enabled.log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf fusion_disabled 5.2742 0.35215 ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_disabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_enabled 5.4487 0.36866 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf fusion_disabled 5.4403 0.36782 ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_disabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_enabled 17342.4348 3703.13932 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf fusion_disabled 18627.0624 3998.42475 ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_disabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_enabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_enabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf fusion_disabled 363.8913 33.14007 ppl-value-checks/gpt-oss-20b-mxfp4.fusion_disabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_enabled 17330.3926 3716.70472 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_enabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf fusion_disabled 17933.9524 3883.17066 ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_disabled.log
2026-06-11 17:23:58 +02:00
6d815ca4d1
Refactor and clean-up host-side fusion logic
2026-06-11 13:46:48 +02:00
c91327f44b
Merge single-lane mm-fusion helpers
2026-06-11 11:30:34 +02:00
9088a3c6d2
Cleanup
2026-06-10 17:36:56 +02:00
096597884b
Restrict lane scale to NVFP4
...
Don't need to test unfused combinations
2026-06-10 17:33:57 +02:00
9d9f1a0086
Reorder scale & bias-add to adhere to #24331
2026-06-10 17:33:57 +02:00
28bde658e9
Restrict scale-fusion to NVFP4
...
This is necessary, as the prolog is quite heavy in GEMV for some
quants/model configs, leading to net perf regression.
We should really be looking to refactor this such that ratio of
prologue/hot-loop/epilogue is better on the hot-loop
front:
+ ./scripts/compare-llama-bench.py -b master -c c1b9381d32c1b9381d3
2026-06-10 17:32:11 +02:00
29713604f1
Refactor mmvq loads, still does not help non-nvfp4 kernels
2026-06-10 15:10:32 +02:00
c1b9381d32
Restrict scale_view_nodes, enroll MM + ADD into lane-matcher
2026-06-10 15:10:32 +02:00
cf97e37d81
Enable fusion for MoE in shared MMVQ
2026-06-10 15:10:32 +02:00
b07d0f03d3
Merge old mul_mat patterns into the lane-based approach
2026-06-10 15:10:32 +02:00
d6190b97a5
Cleanup ggml-cuda/mmvq
...
1. Unrestrict post-scale fusion
2. Rename names accordingly
3. Remove env variable to disable fusion
2026-06-10 15:10:32 +02:00
4ed8ef226f
Cleanup test-backend-ops
2026-06-10 15:10:32 +02:00
2173d97dd4
Add tests for the added fusion ops
2026-06-10 15:10:32 +02:00
45d4704267
Add dense MMVQ fusion as well
...
Perf numbers on B4500. Note qwen35 is FP8->Q8
+ ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite
| Model | Test | t/s master | t/s osimons/nvfp4_fuse_mmvq | Speedup |
|:-------------------------|:-------------|-------------:|------------------------------:|----------:|
| qwen35moe 35B.A3B NVFP4 | tg128@d32768 | 150.15 | 156.29 | 1.04 |
| qwen35moe 35B.A3B Q4_K_M | tg128@d32768 | 157.91 | 157.64 | 1.00 |
Perf numbers on DGX Spark
+ ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite
| Model | Test | t/s master | t/s osimons/nvfp4_fuse_mmvq | Speedup |
|:-------------------------|:-------------|-------------:|------------------------------:|----------:|
| qwen35moe 35B.A3B NVFP4 | tg128@d32768 | 58.31 | 59.69 | 1.02 |
| qwen35moe 35B.A3B Q4_K_M | tg128@d32768 | 54.94 | 54.79 | 1.00 |
2026-06-10 15:10:32 +02:00
f3d3398b97
CUDA: Fuse MMVQ for NVFP4 and BS 1
...
TODO:
1. Add tests to test-backend-ops (did verify correctness manually for
one model)
2. Reorder bias/scale once PRs for NVFP4 are merged/landed
2026-06-10 15:10:32 +02:00
8f83d6c271
mtmd : add video input support ( #24269 )
...
* wip
* ok: lazy bitmap API
* remember to free lazy text
* wip
* add mtmd_helper_video
* support video input on server (base64 input)
* add MTMD_VIDEO config
* add timestamp
* update CLI
* cli: allow auto-completion for video
* add --video arg
* fix build
* update docs
* rename as suggested
2026-06-08 14:40:12 +03:00
c2b1518fd4
sync : ggml
2026-06-08 14:31:33 +03:00
6a1de6fbf1
ggml : bump version to 0.14.0 (ggml/1533)
2026-06-08 14:31:33 +03:00
715b86a366
cli: fix spinner not show during prompt processing ( #24283 )
2026-06-08 11:11:45 +02:00
c74759a244
vulkan: Use cm2 decode_vector for mul_mat_id B matrix loads ( #23991 )
...
This allows vec4 loads of the B elements. Also increase BK to 64 when this is
enabled. Neither of these alone is consistently faster, but together these give
a nice speedup.
In ggml-vulkan.cpp, we need to make sure the B matrix alignment and stride are
multiples of 4.
2026-06-08 10:40:37 +02:00
0f7fada56b
cuda: reset cuda context after reading memory size ( #23935 )
...
* cuda: reset device in get_memory function if no backend is active
* also count device and host buffers
* exclude hip and musa from counting and device reset
* use device mutex instead of atomic
* undo backend_free function move
2026-06-08 10:22:44 +02:00
19bba67c1f
HIP: add gfx1152 and gfx1153 to RDNA3.5 ( #24129 )
2026-06-08 08:33:23 +02:00
daf6bc9f2d
metal : fix im2col 1D case (audio models) ( #24220 )
2026-06-08 09:03:18 +03:00
d403f00ec3
[SYCL] Update compute runtime version to 26.x in docker ( #24070 )
...
* update compute runtime from 25 to 26 in docker
* add comment with old driver for multiple GPUs
2026-06-08 10:35:18 +08:00
9e3b928fd8
common : relax sampler name matching ( #23744 )
...
* common : relax sampler name matching
Currently, in some cases, the alternative names for samplers (like
`top-k` and `min-p` instead of the canonical `top_k` and `min_p`) are
not always recognized by the `common_sampler_types_from_names` function
in `common/sampling.cpp`.
This PR changes the signature of this function to remove the `bool
allow_alt_names` flag, and removes all occurences of the flag from call
sites. Therefore, the function will now always match all known names.
I also changed the logic of the function to unconditionally check the
provided sampler names against both the canonical and alternative names,
and to be case-insensitive.
This fixes an issue I was seeing wherein samplers specified in the
`llama-server` UI were not recognized as valid when the alternative
names were used.
* add more alt names
* cont. fix
* cast to unsigned char for correctness
* common : unify sampler name mapping
* annotate canonical vs. alt sampler name mappings per @CISC
* Update common/sampling.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* common : auto-generate sampler name aliases per @ngxson
* use merged map for matching
* use `.merge` instead of iterating
* nit: simplify comment
* nit: use insert everywhere, not index assignment
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-06-07 22:48:11 +02:00
8a963fc10e
convert : fix conversion for Mistral-Medium-3.5-128B ( #24268 )
...
Mistral explicitly sets `moe` and `llama_4_scaling` to `null` in
params.json, breaking `key in dict` checks during conversion. Replace
with `dict.get(key) is not None` where this matters.
Fixes `convert-hf-to-gguf.py --mistral-format Mistral-Medium-3.5-128B`
2026-06-07 21:41:39 +02:00
379ac6673b
kv-cache : avoid kv cells copies ( #24277 )
2026-06-07 21:42:54 +03:00
f0156d1401
kv-cache: follow the source cache size when sharing cells ( #24267 )
...
A fitted target context can end up smaller than the draft default, the
oversized assistant views then overflow the shared K/V tensors and trip
the ggml_view_4d size assert during graph reserve.
2026-06-07 18:33:00 +03:00
04eb4c446d
llama : add Gemma4 MTP ( #23398 )
2026-06-07 20:50:54 +08:00
8a091c47ab
spec : fix vocab compatibility check ( #24256 )
2026-06-07 14:43:52 +03:00
465b1f0e75
arg: Skip mmproj download when user supplied mmproj ( #24239 )
2026-06-07 11:18:44 +02:00
f71af352a5
convert : fix Gemma4 with no audio encoder ( #24242 )
2026-06-07 08:43:05 +02:00
3f7c79d7b5
docker : bump cuda13 to 13.3.0 ( #24228 )
2026-06-07 08:31:58 +02:00
98d5e8ba8a
common/chat : fix LFM2/LFM2.5 reasoning round-trip and <think> leak ( #24234 )
...
* common/chat : fix LFM2 reasoning round-trip and stray <think> leak
* Gate by reasoning format and whether the template supports <think>
2026-06-06 22:39:21 +02:00
31e82494c0
mtmd: support "frame merge" for qwen-vl-based models ( #21858 )
...
* feat: add video support for Qwen3.5
* various clean up
* revise the design
* fix llava-uhd case
* nits
* nits 2
---------
Co-authored-by: andrewmd5 <1297077+andrewmd5@users.noreply.github.com >
2026-06-06 21:17:25 +02:00
6b80c74f28
completion : remove useless statics ( #24226 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-06 12:16:16 +02:00
588f0dc2ce
completion : fix format specifier in LOG_INF ( #24213 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-06 11:24:27 +02:00
f5c6ae1827
mtmd, server: add "placeholder bitmap" for counting tokens , add */input_tokens API ( #23913 )
...
* mtmd: add "placeholder bitmap" for counting tokens w/o preprocessing
* fast path skip preproc for placeholder
* fix build
* correct the api
* add server endpoint + tests
* add object name
* update docs
* add proxy handling
* fix build
* fix audio input path
* use is_placeholder in process_mtmd_prompt()
* nits
* nits (2)
* docs: clarify chat/completions/input_tokens is not official
* fix merge problem
2026-06-06 11:06:51 +02:00
5a69c97439
vulkan: check coopmat2 features before reporting support ( #24186 )
2026-06-06 09:11:35 +02:00
5343f4502a
model : rename local n_layer_all variable ( #24209 )
2026-06-06 07:07:20 +03:00
603300b008
context : fix off-by-one comparisons to n_gpu_layers ( #24208 )
2026-06-06 07:06:47 +03:00
308f61c31f
opencl: improve get_rows, cpy, concat and q6_k flat gemv ( #24160 )
...
* opencl: allow multiple workgroups for large rows
* opencl: improve small cpy
* opencl: packed concat for small input
* opencl: tweak flat q6_K gemv, increase N_DST and remap threads
2026-06-05 13:45:25 -07:00
da87e9b612
common/chat : unify and fix LFM2/LFM2.5 tool parser ( #24178 )
2026-06-05 14:31:56 -05:00
e82beaa60d
vulkan: add fwht support for Intel with shmem reduction ( #23964 )
...
* vulkan: add fwht support for Intel with shmem reduction
* don't use N as workgroup size
* disable subgroup shuffle on MoltenVK AMD
* disable fwht shader on Intel Windows due to driver bug
2026-06-05 19:44:40 +02:00
c4a278d68e
model: fix build failed ( #24193 )
2026-06-05 18:12:27 +02:00
Oliver Simons