kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
4318037ca3
TensorRT-LLMs/examples/models/core
History
zhhuang-nv 8452775db8
[TRTLLM-5070][feat] Support FP8 KV Cache Reuse for MLA (#4535) 
* optimize kv cache reuse workflow for MLA

write kv cache first and only call up-projection GEMM once
relax contiguous requirements of k/v for setting paged kv cache
return two contiguous tensors when loading MLA KV Cache

Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>

* support fp8 kv cache for MLA kv cache reuse

Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>

* resolve comments

Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>

---------

Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>
2025-05-23 19:47:50 +08:00
..
bert Adding local paths to the datasets to make them loadable in offline mode (#3750) 2025-04-25 03:51:00 +08:00
commandr doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
deepseek_v3 [TRTLLM-5070][feat] Support FP8 KV Cache Reuse for MLA (#4535) 2025-05-23 19:47:50 +08:00
enc_dec chore: Remove deprecated Python runtime benchmark (#4171) 2025-05-14 18:41:05 +08:00
exaone doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
gemma doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
glm-4-9b doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
gpt move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
granite move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
internlm2 doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
llama doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
mamba move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
mixtral move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
mllama doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
multimodal refactor: use x is None instead of x == None. (#4244) 2025-05-15 20:00:04 +08:00
nemotron move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
nemotron_nas doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
phi move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
qwen Breaking change: perf: Enable scheduling overlap by default (#4174) 2025-05-15 14:27:36 +08:00
qwen2audio move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
qwenvl refactor: use x is None instead of x == None. (#4244) 2025-05-15 20:00:04 +08:00
recurrentgemma move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00
vit doc: fix path after examples migration (#3814) 2025-04-24 02:36:45 +08:00
whisper move the reset models into examples/models/core directory (#3555) 2025-04-19 20:48:59 -07:00