kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
635 Commits 25 Branches 59 Tags 2.1 GiB
b91da764de
Commit Graph

635 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yuxianq
b91da764de
chore: remove DummyKvCacheManager. (#3896) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-04-29 09:59:37 +08:00
Jinyang Yuan
dafc28fb85
fix: Fix FMHA-based MLA in the generation phase and add MLA unit test (#3863) 2025-04-29 09:09:43 +08:00
Erin
0577ea0155
waive test_attention_no_cache (#3921) 
Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
 2025-04-28 13:57:01 -07:00
Mike Iovine
e534bf09cc
[fix] Fix flashinfer + speculation issues (#3686) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-28 14:34:22 -04:00
xiweny
f84dd8f815
test: add deepseek v3 & r1 cases (#3528) 
* test: add deepseek v3 & r1 cases

Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
 2025-04-28 23:37:26 +08:00
Yukun He
5502a522d2
Fixing minor typo in allreduce kernel selection (#3912) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
Co-authored-by: Kefeng-Duan <176893526+Kefeng-Duan@users.noreply.github.com>
 2025-04-28 23:06:49 +08:00
Mike Iovine
e6f7ff3a46
[chore] Make llama4 MoE use maybe_execute_in_parallel (#3779) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-28 10:58:03 -04:00
Zhenhuan Chen
19da82d68f
fix(requirements): fix neither 'setup.py' nor 'pyproject.toml' found (#3906) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-04-28 18:35:19 +08:00
Xianjie Qiao
3617e948fd
Add docs about DeepSeek-R1 long context support. (#3910) 
* Add docs about DeepSeek-R1 long context support

Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

* update docs

Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

* reformat

Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>

---------

Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
 2025-04-28 18:33:05 +08:00
Zhenhuan Chen
ad15e45f07
[TRTLLM-4638 ][feat] add best of n support with reward model in scaffolding (#3807) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-04-28 17:15:33 +08:00
Tao Li @ NVIDIA
2fe35924e3
Fix the link of doc (#3903) 
Signed-off-by: taoli <litaotju@users.noreply.github.com>
Co-authored-by: taoli <litaotju@users.noreply.github.com>
 2025-04-28 14:41:40 +08:00
xinhe-nv
82a8e43557
test: [CI] Add failed cases into waives.txt (#3867) 
* update waive list

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

* update waives

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

---------

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-28 14:32:48 +08:00
xinhe-nv
e20b67e9fd
update waives & tests (#3887) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-28 14:29:35 +08:00
Zhenhuan Chen
d5bca18807
infra: add scaffolding paths to pytorch only files (#3835) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-04-28 13:49:27 +08:00
Yanchao Lu
068c72ebf8
Test: waive intermittent test hang (#3894) 
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-04-28 08:53:20 +08:00
bhsueh_NV
f77252e9ff
fix bug of create cuda stream as default parameter which will be init… (#3764) 
* fix bug of create cuda stream as default parameter which will be initialized during importing

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* add torch.cuda.Stream() for the leader node

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix pre-commit issue

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-28 08:16:03 +08:00
Iman Tabrizian
74cc9e26ff
infra: install Triton in the base image (#3759) 
* infra: install Triton in the base image

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

* install Triton from the base image

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

* update base image

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

* Address review comments

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

* update base image

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

* waive test

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

---------

Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-04-28 07:36:30 +08:00
Yan Chunwei
ad4226d946
fix: trtllm-bench build trt engine on slurm (#3825) 
* add submit_sync to RemoteMpiSessionClient

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

add barrier

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

fix comment

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

disable test

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* fix

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-27 22:26:23 +08:00
bhsueh_NV
76f2c631fb
fix: add warmup flag into py_executor to prevent enable profiler during wa… (#3852) 
* add warmup flag into py_executor to prevent enable profiler during warmup

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix bug of pre-commit

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* change setting warmup to all ranks

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-27 19:22:42 +08:00
Chuang Zhu
e2318756ed
cacheTransceiver buffer manager (#3798) 
* cacheTransceiver buffer manager

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

* fix args

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

* cpp kvCacheManager

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

* format

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

---------

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-04-27 11:48:15 +08:00
HuiGao-NV
136aab5c54
fix: Update num_of_ctx_tokens in iteration stats (#3785) 
* Update num_of_ctx_tokens in iteration stats
* Revert not neccessary change of importing module
 2025-04-27 10:24:47 +08:00
Emma Qiao
a4b483b969
Infra: Remove empty junit xml (#3794) 
* Remote results.xml when no cases ran

Signed-off-by: qqiao <qqiao@nvidia.com>

* Change some test config to verify

Signed-off-by: qqiao <qqiao@nvidia.com>

* Update for quotes

Signed-off-by: qqiao <qqiao@nvidia.com>

* Move the remove results.xml in catch section

Signed-off-by: qqiao <qqiao@nvidia.com>

* Add missed path

Signed-off-by: qqiao <qqiao@nvidia.com>

* Change back the test stage setting

Signed-off-by: qqiao <qqiao@nvidia.com>

---------

Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-04-26 18:46:18 -07:00
bhsueh_NV
e9fab4f3d9
fix bug of deepseek gropu_size setting (#3860) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-27 09:10:37 +08:00
yuxianq
e6c14ca97a
fix: Detect pmix and raise error when mpirun is not used. (#3858) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-04-26 21:49:41 +08:00
milesial
362a8272f8
feat: llama4 input processor (#3383) 
Signed-off-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com>
Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
Co-authored-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com>
Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
 2025-04-25 16:47:14 -07:00
Kaiyu Xie
d7472231f9
TRTLLM-4875 feat: Add version switcher to doc (#3846) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-26 05:42:46 +08:00
Dom Brown
7ff9fd345c
Test: Split C++ unit tests for CI granularity (#3868) 
Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-04-25 13:30:58 -07:00
QI JUN
6ac1a54f57
chore: update pytorch only change file list (#3873) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-26 04:03:51 +08:00
qixiang-99
ecd621fb0a
feat: Add head size 72 support for QKV Preprocessing kernel (#3743) 
* refactor: Fix headsize 72 attention error for TRTLLM attn backend in PyTorch workflow

- Remove the head size pre-check logic in AttentionOp because head size 72 can be supported with fmha kernels.
- Added support for head size 72 in unfused attention kernels(QKVPreprocessing).
- Enhanced unit tests by introducing a scenario generation function for better test coverage of attention configurations(include head size 72).

Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>

* update: Waive head_dim=72 test cases and enhance test representation

- Added a waiver for head_dim=72 cases on post sm100 in the test suite to address known issues.
- Introduced a custom __repr__ method in the Scenario class for pytest substring match.

Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>

---------

Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>
 2025-04-25 11:07:40 -07:00
sugunav14
5b9897a8cd
fix: [AutoDeploy] update hf loading for e_score_correction_bias (#3847) 
Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
 2025-04-26 02:03:47 +08:00
Mike Iovine
68e774ff9e
[chore] Add Llama 4 Maverick to quickstart README (#3848) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-26 01:04:24 +08:00
Yiqing Yan
238fefc659
[infra] Waive L0 tests (#3853) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-25 17:32:21 +08:00
dongxuy04
16535991b2
feat: Add MNNVL MoE A2A support (#3504) 
* add MNNVL memory mapping support

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add more MPI environment for trtllm-llmapi-launch

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add MoE communication and prepare kernels

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add MNNVL AlltoAll support for DeepSeekV3

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add output dump for throughput benchmark

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* support dynamic kernel launch grid

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* address review comments

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* address review comments #2

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

---------

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
 2025-04-25 17:29:08 +08:00
Yuan Tong
57944206ba
feat: return logits in PyTorch flow (#3221) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-24 16:56:03 -07:00
QI JUN
991939a0f4
chore: increase A30 for cpp test (#3811) 
* increase A30 for cpp test

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* enable parallel run test for gpt_executor

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* clean

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* decrease freeGpuMemoryFraction of cpp tests

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* fix

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

---------

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-24 16:34:39 -07:00
hlu1
d72add1794
[Deepseek] Pass hidden_states_fp4 to shared_experts (#3819) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com@users.noreply.github.com>
 2025-04-24 13:12:12 -07:00
rakib-hasan
ccd1eb67ec
Adding local paths to the datasets to make them loadable in offline mode (#3750) 
* adding local paths to the datasets to make them loadable in offline mode

Signed-off-by: Rakib Hasan <rhasan@nvidia.com>

* bert datasets should work on both offline and online mode

Signed-off-by: Rakib Hasan <rhasan@nvidia.com>

---------

Signed-off-by: Rakib Hasan <rhasan@nvidia.com>
 2025-04-25 03:51:00 +08:00
Luis Vega
f95dbbb6cb
added nemotron-h to supported models (#3663) 
Signed-off-by: Luis Vega <lvega@nvidia.com>
 2025-04-24 10:41:32 -07:00
HuiGao-NV
7420ddc3d0
fix: fix lora case failure (#3838) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-04-24 07:29:08 -07:00
WeiHaocheng
3fc2a16920
feat(part 2): Enhance the integrated robustness of scaffolding with __init__.py #3305 (#3731) 
Signed-off-by: fredw (generated by with_the_same_user script) <20514172+WeiHaocheng@users.noreply.github.com>
 2025-04-24 18:47:03 +08:00
Zhanrui Sun
ae34d60108
chore: bump version to 0.20.0rc1 (#3834) 
Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-04-24 17:43:37 +08:00
Shi Xiaowei
1d5178814b
Fix: Revert commit 25f9669 (#3832) 
Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>
 2025-04-24 14:03:20 +08:00
qixiang-99
3f67a4c9d8
fix: Set default prompts and media for multimodal quickstart example (#3792) 
Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>
 2025-04-23 22:02:28 -07:00
Enwei Zhu
777c40e5fa
[https://nvbugspro.nvidia.com/bug/5238599][fix] Normalize example path in accuracy tests 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-24 10:09:59 +08:00
Enwei Zhu
90b708f851
[https://nvbugspro.nvidia.com/bug/5238602][fix] Package lm_eval configuration files 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-24 09:57:01 +08:00
xinhe-nv
476d7003f8
test: [CI] Add failed cases into waives.txt (#3777) 
* update waive list

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

* update waives.txt

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

---------

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-24 09:36:05 +08:00
hlu1
cd2bcdc1a9
Fix create_weights in attention (#3692) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>
 2025-04-24 07:30:00 +08:00
Mike Iovine
bc5fe7800d
[chore] Fix KV cache block reuse flag name in quickstart_advanced (#3781) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-24 06:02:47 +08:00
QI JUN
d0d19e81ca
chore: fix some invalid paths of contrib models (#3818) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-24 05:36:16 +08:00
Kaiyu Xie
dfbcb543ce
doc: fix path after examples migration (#3814) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-24 02:36:45 +08:00