kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-13 22:18:36 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
539 Commits 25 Branches 59 Tags 2.1 GiB
c6f7d42335
Commit Graph

539 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Zac Patel
c6f7d42335
doc: Release V0.19 Perf Overview Update (#4166) 
Signed-off-by: zpatel <22306219+zbpatel@users.noreply.github.com>
 2025-05-09 18:02:35 +08:00
nv-guomingz
9727aaec9a
docs:add torch flow supported model list. (#4129) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-05-08 10:25:58 +08:00
nv-guomingz
00955ce56e
docs:update 0.19 doc. (#4120) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-05-07 19:49:30 +08:00
Kaiyu Xie
50b5b1ab5b
doc: update release notes (#4108) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-07 14:01:16 +08:00
Kaiyu Xie
faf839f100
Update switcher (#4098) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-06 23:50:03 +08:00
Yukun He
c11fc81184
Optimize the AutoTuner cache access code to reduce host code overhead. (#4060) 
The NVFP4 Linear op is very sensitive to the host overhead.
This PR introduces customizable `find_nearest_profile` and `get_cache_key_specifc`, which allow users to override the default method for generating the cache key.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-05-06 10:57:41 +08:00
Kaiyu Xie
eabf716ff2
doc: Update 0.19.0 release notes (#3976) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-06 08:59:07 +08:00
Yan Chunwei
090d207ac1
fix waive (#4048) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-05-05 13:17:29 +08:00
Yukun He
b051a8f5ce
Fix the perf regression caused by insufficient cache warmup. (#4042) 
Force tuning up to 8192 sequence length for NVFP4 linear op. Also, make this runtime-selectable with UB enabled.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-05-04 00:46:23 +08:00
jiahanc
51d25c622e
Doc: Fix H200 DeepSeek R1 perf doc (#4006) 
* fix doc

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>

* update perf number

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>

---------

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
 2025-05-02 09:01:07 +08:00
Enwei Zhu
dbcfd78ea8
[https://nvbugspro.nvidia.com/bug/5247148][fix] Attention DP with overlap scheduler (#3975) 
* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* update multigpu list

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix namings

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-05-01 10:49:45 +08:00
Anurag Mukkara
07479eabc7
Add more valid outputs (#4004) 
Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
 2025-04-30 17:47:55 -07:00
nv-guomingz
034f6f2d91
docs:update 0.19 docs (#3986) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-30 19:25:26 +08:00
Tracin
cd9c7498d0
Fix mixtral fp4 llmapi bug. (#3929) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-30 14:56:57 +08:00
bhsueh_NV
de774ab283
fix bug of create cuda stream as default parameter which will be init… (#3764) (#3901) 
* fix bug of create cuda stream as default parameter which will be initialized during importing



* add torch.cuda.Stream() for the leader node



* fix pre-commit issue



---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-30 12:25:26 +08:00
Ivy Zhang
b29f427566
skip fp8 gemm for pre-hopper (#3931) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-30 12:22:19 +08:00
Yan Chunwei
be61b4d9f8
fix: remote mpi session abort (#3884) 
* fix remote mpi session

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* fix

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-29 18:17:31 +08:00
nv-guomingz
9fe579aa01
docs:fix https://nvbugs/5244616 by removing new invalid links. (#3939) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-29 16:49:16 +08:00
Tao Li @ NVIDIA
98a9dfca97
fix(requirements): fix neither 'setup.py' nor 'pyproject.toml' found (#3906) (#3926) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
Co-authored-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-04-29 09:55:13 +08:00
nv-guomingz
03dad6287a
docs:fix https://nvbugs/5244616 by removing invalid links. (#3918) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-29 07:58:52 +08:00
Yan Chunwei
d244d7c891
waive a test (#3897) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-28 13:12:48 +08:00
Yiqing Yan
9f1305edf7
[Infra] Waive l0 test (#3895) 
* Waive L0 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* Waive L0 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

---------

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-28 11:00:39 +08:00
Yan Chunwei
c691477092
waive a test (#3886) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-27 16:47:18 +08:00
Kaiyu Xie
fde74feeb0
TRTLLM-4875 feat: Add version switcher to doc (#3871) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-26 07:18:49 +08:00
Enwei Zhu
29344e4697
[https://nvbugspro.nvidia.com/bug/5241495][fix] CUDA Graph padding with overlap scheduler (#3839) 
* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fuse

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-25 16:56:48 +08:00
Frank
7f8f227a13
fix: Add pre-download of checkpoint before benchmark. (#3772) 
* Add pre-download of checkpoint before benchmark.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Add missing remote code flag.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Move from_pretrained to throughput benchmark.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Move download and use snapshot_download.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Removed trusted flag.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Fix benchmark command in iteration log test.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

---------

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-04-25 09:02:53 +08:00
bhsueh_NV
ae910e9f00
fix: Fix/fused moe 0.19 (#3799) 
* fix bug of stream init

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix bug

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-24 22:25:41 +08:00
wili
33c4d49a77
[doc] Better document for Draft-Target-Model (DTM) speculative decoding (#3797) 
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-04-24 11:41:35 +08:00
Yukun He
ab2f663101
fix: Reduce memory usage in fused moe op associated with AutoTuning and fix moe fallback issue. (#3793) 
* Reduce memory usage in fused moe op associated with AutoTuning.
* Replace pre-defined bucket size strategy with a generating function based on the tune_max_num_tokens.
* Add free_memory logic of workspace in min_latency_mode fused moe path.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

* Fix fused_moe fallback issue. (#3652)

min_latency_mode is only set to False during warmup phase. Thus when it becomes true during inference, all tactics fall back to the default one and thus cause perf regression.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

---------

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-24 10:14:26 +08:00
Yiqing Yan
e0691e6e27
Waive L0 tests (#3826) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-24 10:11:21 +08:00
Barry Kang
41b03714cd
Fix ModelOpt Mixtral AWQ OOM (#3714) (#3761) 
Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-24 03:17:48 +08:00
Fanrong Li
2f02263912
add know issue to deepseek doc. (#3800) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-23 17:21:50 +08:00
Ivy Zhang
f056d440ab
test: [nvbug: 5234494] skip_pre_ada for fp8 cases (#3718) 
* skip_pre_ada for fp8 cases

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* update

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* update after rebase

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

---------

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-23 16:54:40 +08:00
xinhe-nv
b0ac7c9ea9
test: skip failed cases on B200 (#3710) 
* add skip condition to tests

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

* fix error

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

---------

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-23 16:19:39 +08:00
Chuang Zhu
f08e59940b
skip disagg deepseek test if sm!=90 (#3720) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-04-23 14:47:45 +08:00
Ivy Zhang
3e56e402a5
remove benchmark test list (#3643) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-23 13:30:09 +08:00
Anurag Mukkara
851e2f5f9e
fix: PP4 fixes and cleanup (#3688) 
Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>
 2025-04-23 10:53:54 +08:00
Yukun He
a824946622
https://nvbugs/5141291: Fix convert.py script for Qwen model. (#3758) 
Include Qwen2VLDecoderLayer in the smooth_qwen2_model function.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-22 20:18:03 +08:00
Ivy Zhang
b11cb2fdd0
waive gemma on L20 (#3766) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-22 18:48:06 +08:00
dominicshanshan
792b71f412
Update ds v3 parameters in stress test. (#3676) 2025-04-22 18:04:17 +08:00
ruodil
793d0102d6
waive failed case in perf test, change default max_batch_size to 512 and write config.json to output log (#3656) 
Signed-off-by: Ruodi <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-22 14:53:21 +08:00
Yiqing Yan
52e6702725
Waive L0 test (#3756) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-22 14:26:19 +08:00
YueWeng
611ef8eb9d
test: Get Eagle tests working (#3593) (#3722) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
 2025-04-22 13:59:23 +08:00
nv-guomingz
ba15155a9c
doc:add torch examples link into torch backend documentation (#3749) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-21 22:24:10 -07:00
peaceh-nv
e19309c98e
chore : Split more tests out of gpt tests (#3524) (#3674) 
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
 2025-04-21 20:19:40 -07:00
Yan Chunwei
e69d7bb1ad
unwaive multi-node test (#3721) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-22 07:56:29 +08:00
Tracin
81d1f4f1c0
Fix script options for engines. (#3622) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-21 18:59:18 +08:00
DylanChen-NV
8a8a55a2f0
fix: FP8 kv accuracy (#3675) 
* fix FP8 kv accuracy

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

* update doc

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

---------

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
 2025-04-21 15:59:31 +08:00
Tracin
8f17f3fa1d
Un-waive DS-V3-Lite tests. (#3621) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-21 14:54:24 +08:00
Yibin Li
b3ce6380cd
Fix: cherry-pick hmac encryption from main branch (#3635) 
* security fix cherry-pick changes from main

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>

* fix hmac in remote mpi session (#3649)

Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>
Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
 2025-04-21 13:08:03 +08:00