kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
467 Commits 25 Branches 59 Tags 2.1 GiB
fffb403125
Commit Graph

467 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Robin Kobus
fffb403125
fix: disable KV cache reuse if using attention sink (#3021) 
* fix: disable KV cache reuse if using attention sink

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* fix: disable KV cache reuse if sink bubble

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* add comment

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

---------

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-04-16 03:07:32 +08:00
Pengyun Lin
1899e71364
doc: add genai-perf benchmark & slurm multi-node for trtllm-serve doc (#3407) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-04-16 00:11:58 +08:00
Kaiyu Xie
e037d3e99b
chore: Unify Python NVTX call (#3450) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-15 23:25:36 +08:00
Kaiyu Xie
258ae9c58c
Revert "infra: move nvrtc_wrapper to conan (#3282)" (#3573) 
This reverts commit c0dd6cbce0.

Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-15 22:45:13 +08:00
HuiGao-NV
d35db254e2
test: Enable 4 multi-gpu test cases for deepseek (#3569) 
Signed-off-by: Hui Gao <huig@nvidia.com>
Signed-off-by: Hui Gao†<huig@nvidia.com>
 2025-04-15 22:01:52 +08:00
Yan Chunwei
c27e130be0
unwaive test (#3559) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-15 19:42:06 +08:00
jiahanc
1d3b98b920
perf: Optimize quantization kernels used in DeepSeek on Hopper (#3466) 
Signed-off-by: jiahanc <jiahanc@nvidia.com>
 2025-04-15 17:49:57 +08:00
xinhe-nv
5cfa927132
update waive list (#3503) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-15 16:53:53 +08:00
bhsueh_NV
3aa37e6b72
fix bug (#3570) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-15 16:50:22 +08:00
Yuan Tong
d4c0423cdb
refactor: collect executor and decoder states into dataclass (#3234) 
* fix: Proper error bubbling for PyExecutor

Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-15 16:31:45 +08:00
Robin Kobus
b7a38feb14
chore: Clean up cpp runtime (#3537) 
* add space in test output

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* perf: reduce executor lock scope

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: Move TokenRangeRetentionConfig implementation to cpp file

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* fix: Improve finished steps handling for external draft tokens

- Fixed a bug where the whole finished steps tensor was being zeroes instead of the slices.
- Replaced the creation of a temporary tensor for finished steps with a direct slice from the input tensor, improving efficiency and readability.
- Updated the tensor management logic to streamline the process of setting zero values for finished steps during batch processing.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* chore: Clean up includes

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

---------

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-04-15 16:06:14 +08:00
shaharmor98
ede7058544
Feat/ Integrate peftCacheManager in PyExecutor creation (#3372) 
* integrate peftCacheManager in PyExecutor creation

Signed-off-by: Shahar Mor <smor@nvidia.com>
 2025-04-15 15:14:43 +08:00
hlu1
5881a65374
Fix test_fp4_quantize_gemm_torch (#3551) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>
 2025-04-14 23:58:31 -07:00
Yuan Tong
668a0335e4
fix: Proper error bubbling for PyExecutor (#3321) 
* fix: Proper error bubbling for PyExecutor
* fix: Proper shutdown
* fix: multi gpu proper shutdown

Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-04-15 14:49:46 +08:00
xinhe-nv
0e152910f5
update waive list (#3498) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-15 14:33:49 +08:00
Yukun He
cfc6f242dd
Chore: Remove profile test. (#3565) 
Because it is duplicated with test_fp4_linear. Also, cpp profiler has been unified with the new AutoTuner already.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-14 23:17:51 -07:00
Jinyang Yuan
0305942808
chore: Modifications that should have been included but were mistakenly overwritten in PR #3467 (#3557) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-04-15 14:08:07 +08:00
nv-guomingz
39bdb1fe1c
docs:update llm api examples and customizations sections' links. (#3566) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-15 13:55:22 +08:00
yuxianq
0e7e949feb
refactor: Split llama4 model from llama model. (#3530) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-04-15 13:41:05 +08:00
tburt-nv
e1e068d4f3
fix local user (#3550) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
 2025-04-15 13:20:34 +08:00
Bo Li
5eae397b3b
doc: Update instructions to enable FP8 MLA for Deepseek. (#3488) 
* doc: Update doc to enable FP8 MLA for Deepseek.

Signed-off-by: Bo Li <bobboli0202@gmail.com>

* Update.

Signed-off-by: Bo Li <bobboli0202@gmail.com>

* Update.

Signed-off-by: Bo Li <bobboli0202@gmail.com>

* Update the status on Hopper and Blackwell.

Signed-off-by: Bo Li <bobboli0202@gmail.com>

* Update.

Signed-off-by: Bo Li <bobboli0202@gmail.com>

* Update table of contents.

Signed-off-by: Bo Li <bobboli0202@gmail.com>

---------

Signed-off-by: Bo Li <bobboli0202@gmail.com>
Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>
 2025-04-15 13:12:33 +08:00
Zheng Duan
b0cb963199
test: torch-flow conditional disagg test (#3410) 
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
 2025-04-15 10:54:14 +08:00
Jinyang Yuan
175adb94ab
chore: Log memory sizes of weights and activations separately (#3467) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-04-15 09:48:35 +08:00
nv-guomingz
b32ae7ac92
test:add fp8_kv_cache functionality test case. (#3457) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-15 09:16:46 +08:00
QI JUN
112f716155
chore: move all distributed related codes into _torch.distributed directory (#3511) 
* move all distributed related codes into _torch.distributed directory

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* fix ci

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* fix ci

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

---------

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-15 08:39:17 +08:00
brb-nv
098ca7f68c
test: Fix breaking Phi3 multimodal tests (#3544) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
 2025-04-15 08:02:34 +08:00
Iman Tabrizian
bad55e99bb
test: Add MTP + overlap + Attention DP disaggregated test (#3542) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-04-15 07:46:03 +08:00
Pamela Peng
6cdfc54883
feat: Add FP8 support for SM 120 (#3248) 
* Allow FP8 on SM120

Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>

* fix sm121

Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>

* fix

Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>

* fix pre-commit

Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>

* review update

Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>

---------

Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>
 2025-04-14 16:05:41 -07:00
tburt-nv
c0dd6cbce0
infra: move nvrtc_wrapper to conan (#3282) 
* add pip scripts dir to path
* move nvrtc_wrapper to conan
* support building nvrtc wrapper from source

---------

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
 2025-04-15 05:31:01 +08:00
Aurelien Chartier
8cf2785bc6
chore: unify pp_layers helpers (#3429) 
* chore: unify pp_layers helpers

Fix assumptions about equal number of layers per PP rank
in prepare_attention_inputs

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
 2025-04-15 04:49:17 +08:00
Chang Liu
01cb3ccb04
use global expert idx to load expert weights (#3386) 
Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com>
 2025-04-14 11:58:30 -07:00
Chang Liu
1902d73eb5
fix: llama4: add an option apply_router_weight_on_input for in FusedMoE (#3492) 
* apply a tenative fix to moe bypass kernel update

* Pass none to disable final stage in moe

Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com>
Signed-off-by: Chang Liu <lc9114@gmail.com>

---------

Signed-off-by: Chang Liu <lc9114@gmail.com>
Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com>
 2025-04-14 11:56:42 -07:00
Kaiyu Xie
b286b51118
feat: Support torch profiler (#3470) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-14 22:06:06 +08:00
Yuan Tong
5985d362a9
fix: install RTC headers with linking when using --linking_install_binary (#3484) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-04-14 21:22:03 +08:00
Zhanrui Sun
714ff3eedd
chore: bump version to 0.19.0rc0 (#3535) 
Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-04-14 18:11:20 +08:00
Robin Kobus
f58d4698c8
chore: Clean up cpp runtime (#3505) 
* chore: Remove unused tensors from DecoderBuffers

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* fix: Remove unused argument from readme

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* chore: remove unused tensor

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: Remove unnecessary newOutputTokens

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: Remove unnecessary event in getDecoderSlotHostOutputs

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

---------

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-04-14 18:00:03 +08:00
Zhanrui Sun
ee4ce0379d
chore: bump version to 0.19.0rc0 (#3514) 
* chore: bump version to 0.19.0.rc0

Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>

* Update README

Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>

---------

Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-04-14 17:32:30 +08:00
Ivy Zhang
170bc22139
fix test name (#3534) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-14 17:09:50 +08:00
Zhenhuan Chen
c0493523d0
infra: add more codespell ignore words, prevent ans->and (#3513) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-04-14 17:00:30 +08:00
Kaiyu Xie
f99be2726f
doc: Add example section for multi-node DeepSeek R1 benchmark on GB200 (#3519) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-14 16:45:55 +08:00
Chuang Zhu
2e669133c2
disagg perf tune doc (#3516) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-04-14 16:05:06 +08:00
xinhe-nv
b1d8495b3d
update waive list (#3510) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-14 15:24:48 +08:00
dongjiyingdjy
2fb1d65d43
fix: fix max_seq_len in executor_config (#3487) 
Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
 2025-04-14 15:13:29 +08:00
HuiGao-NV
9f41e826bf
fix: remove one duplicated line of code (#3523) 
Signed-off-by: Hui Gao <huig@nvidia.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-14 14:52:46 +08:00
bhsueh_NV
9d7d48faeb
fix: disable the kv cache reuse for prompt tuning test (#3474) 
* disable the kv cache reuse for prompt tuning test

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* unwaive the wavied tests

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-14 14:35:47 +08:00
brb-nv
44090a5388
Add support for Phi-4-MM (#3296) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-04-14 14:24:10 +08:00
QI JUN
8d3e449a8d
reduce num layers in attention test (#3509) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-14 12:43:59 +08:00
Yiqing Yan
19d296b4b2
chore: add dgx_h200 tests (#3451) 
* add dgx_h200 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* test

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* fix pre-commit

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* fix

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* fix

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* change bsl branch

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* fix

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* change multi gpu related file list

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

---------

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-14 11:20:55 +08:00
yuxianq
9d64b6b890
Cache sin cos in model instead of global LRU cache. (#3378) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-04-14 11:19:09 +08:00
pcastonguay
fe6f14b2b1
fix: Fixing issue with first gen token being returned twice in streaming (#3427) 
* fix: Fixing issue with first gen token being returned twice with streaming

Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>

* Fixing not_expectring_strings in test

Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>

---------

Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-13 22:45:09 -04:00