kanshan/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-07-04 18:20:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

model : NvFP4 quantized LM head support (#23046)

Browse Source 
* NvFP4 quantized LM head support

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review commnets

Signed-off-by: ynankani <ynankani@nvidia.com>

* Add assert for NvFp4 lm head and tied embeddings

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review commnets

Signed-off-by: ynankani <ynankani@nvidia.com>

* Create output_s tensor only when LM head NvFp4

Signed-off-by: ynankani <ynankani@nvidia.com>

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
This commit is contained in:
ynankani
2026-05-16 09:09:27 +00:00
committed by GitHub
parent 59778f0196
commit 42928bc14d
103 changed files with 121 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/models/phi3.cpp
+1 -1
View File
@@ -179,7 +179,7 @@ llama_model_phi3::graph<iswa>::graph(const llama_model & model, const llm_graph_
cb(cur, "result_norm", -1);
res->t_embd = cur;
cur = build_lora_mm(model.output, cur);
cur = build_lora_mm(model.output, cur, model.output_s);
if (model.output_b != nullptr) {
cb(cur, "result_output_no_bias", -1);