kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-13 22:18:36 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
539 Commits 25 Branches 59 Tags 2.1 GiB
release/0.19
Commit Graph

539 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Zac Patel
c6f7d42335
doc: Release V0.19 Perf Overview Update (#4166) 
Signed-off-by: zpatel <22306219+zbpatel@users.noreply.github.com>
 2025-05-09 18:02:35 +08:00
nv-guomingz
9727aaec9a
docs:add torch flow supported model list. (#4129) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-05-08 10:25:58 +08:00
nv-guomingz
00955ce56e
docs:update 0.19 doc. (#4120) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-05-07 19:49:30 +08:00
Kaiyu Xie
50b5b1ab5b
doc: update release notes (#4108) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-07 14:01:16 +08:00
Kaiyu Xie
faf839f100
Update switcher (#4098) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-06 23:50:03 +08:00
Yukun He
c11fc81184
Optimize the AutoTuner cache access code to reduce host code overhead. (#4060) 
The NVFP4 Linear op is very sensitive to the host overhead.
This PR introduces customizable `find_nearest_profile` and `get_cache_key_specifc`, which allow users to override the default method for generating the cache key.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-05-06 10:57:41 +08:00
Kaiyu Xie
eabf716ff2
doc: Update 0.19.0 release notes (#3976) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-06 08:59:07 +08:00
Yan Chunwei
090d207ac1
fix waive (#4048) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-05-05 13:17:29 +08:00
Yukun He
b051a8f5ce
Fix the perf regression caused by insufficient cache warmup. (#4042) 
Force tuning up to 8192 sequence length for NVFP4 linear op. Also, make this runtime-selectable with UB enabled.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-05-04 00:46:23 +08:00
jiahanc
51d25c622e
Doc: Fix H200 DeepSeek R1 perf doc (#4006) 
* fix doc

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>

* update perf number

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>

---------

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
 2025-05-02 09:01:07 +08:00
Enwei Zhu
dbcfd78ea8
[https://nvbugspro.nvidia.com/bug/5247148][fix] Attention DP with overlap scheduler (#3975) 
* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* update multigpu list

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix namings

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-05-01 10:49:45 +08:00
Anurag Mukkara
07479eabc7
Add more valid outputs (#4004) 
Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
 2025-04-30 17:47:55 -07:00
nv-guomingz
034f6f2d91
docs:update 0.19 docs (#3986) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-30 19:25:26 +08:00
Tracin
cd9c7498d0
Fix mixtral fp4 llmapi bug. (#3929) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-30 14:56:57 +08:00
bhsueh_NV
de774ab283
fix bug of create cuda stream as default parameter which will be init… (#3764) (#3901) 
* fix bug of create cuda stream as default parameter which will be initialized during importing



* add torch.cuda.Stream() for the leader node



* fix pre-commit issue



---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-30 12:25:26 +08:00
Ivy Zhang
b29f427566
skip fp8 gemm for pre-hopper (#3931) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-30 12:22:19 +08:00
Yan Chunwei
be61b4d9f8
fix: remote mpi session abort (#3884) 
* fix remote mpi session

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* fix

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-29 18:17:31 +08:00
nv-guomingz
9fe579aa01
docs:fix https://nvbugs/5244616 by removing new invalid links. (#3939) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-29 16:49:16 +08:00
Tao Li @ NVIDIA
98a9dfca97
fix(requirements): fix neither 'setup.py' nor 'pyproject.toml' found (#3906) (#3926) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
Co-authored-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-04-29 09:55:13 +08:00
nv-guomingz
03dad6287a
docs:fix https://nvbugs/5244616 by removing invalid links. (#3918) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-29 07:58:52 +08:00
Yan Chunwei
d244d7c891
waive a test (#3897) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-28 13:12:48 +08:00
Yiqing Yan
9f1305edf7
[Infra] Waive l0 test (#3895) 
* Waive L0 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* Waive L0 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

---------

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-28 11:00:39 +08:00
Yan Chunwei
c691477092
waive a test (#3886) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-27 16:47:18 +08:00
Kaiyu Xie
fde74feeb0
TRTLLM-4875 feat: Add version switcher to doc (#3871) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-26 07:18:49 +08:00
Enwei Zhu
29344e4697
[https://nvbugspro.nvidia.com/bug/5241495][fix] CUDA Graph padding with overlap scheduler (#3839) 
* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fuse

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-25 16:56:48 +08:00
Frank
7f8f227a13
fix: Add pre-download of checkpoint before benchmark. (#3772) 
* Add pre-download of checkpoint before benchmark.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Add missing remote code flag.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Move from_pretrained to throughput benchmark.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Move download and use snapshot_download.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Removed trusted flag.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Fix benchmark command in iteration log test.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

---------

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-04-25 09:02:53 +08:00
bhsueh_NV
ae910e9f00
fix: Fix/fused moe 0.19 (#3799) 
* fix bug of stream init

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix bug

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-24 22:25:41 +08:00
wili
33c4d49a77
[doc] Better document for Draft-Target-Model (DTM) speculative decoding (#3797) 
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-04-24 11:41:35 +08:00
Yukun He
ab2f663101
fix: Reduce memory usage in fused moe op associated with AutoTuning and fix moe fallback issue. (#3793) 
* Reduce memory usage in fused moe op associated with AutoTuning.
* Replace pre-defined bucket size strategy with a generating function based on the tune_max_num_tokens.
* Add free_memory logic of workspace in min_latency_mode fused moe path.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

* Fix fused_moe fallback issue. (#3652)

min_latency_mode is only set to False during warmup phase. Thus when it becomes true during inference, all tactics fall back to the default one and thus cause perf regression.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

---------

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-24 10:14:26 +08:00
Yiqing Yan
e0691e6e27
Waive L0 tests (#3826) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-24 10:11:21 +08:00
Barry Kang
41b03714cd
Fix ModelOpt Mixtral AWQ OOM (#3714) (#3761) 
Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-24 03:17:48 +08:00
Fanrong Li
2f02263912
add know issue to deepseek doc. (#3800) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-23 17:21:50 +08:00
Ivy Zhang
f056d440ab
test: [nvbug: 5234494] skip_pre_ada for fp8 cases (#3718) 
* skip_pre_ada for fp8 cases

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* update

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* update after rebase

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

---------

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-23 16:54:40 +08:00
xinhe-nv
b0ac7c9ea9
test: skip failed cases on B200 (#3710) 
* add skip condition to tests

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

* fix error

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

---------

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-23 16:19:39 +08:00
Chuang Zhu
f08e59940b
skip disagg deepseek test if sm!=90 (#3720) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-04-23 14:47:45 +08:00
Ivy Zhang
3e56e402a5
remove benchmark test list (#3643) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-23 13:30:09 +08:00
Anurag Mukkara
851e2f5f9e
fix: PP4 fixes and cleanup (#3688) 
Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>
 2025-04-23 10:53:54 +08:00
Yukun He
a824946622
https://nvbugs/5141291: Fix convert.py script for Qwen model. (#3758) 
Include Qwen2VLDecoderLayer in the smooth_qwen2_model function.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-22 20:18:03 +08:00
Ivy Zhang
b11cb2fdd0
waive gemma on L20 (#3766) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-22 18:48:06 +08:00
dominicshanshan
792b71f412
Update ds v3 parameters in stress test. (#3676) 2025-04-22 18:04:17 +08:00
ruodil
793d0102d6
waive failed case in perf test, change default max_batch_size to 512 and write config.json to output log (#3656) 
Signed-off-by: Ruodi <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-22 14:53:21 +08:00
Yiqing Yan
52e6702725
Waive L0 test (#3756) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-22 14:26:19 +08:00
YueWeng
611ef8eb9d
test: Get Eagle tests working (#3593) (#3722) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
 2025-04-22 13:59:23 +08:00
nv-guomingz
ba15155a9c
doc:add torch examples link into torch backend documentation (#3749) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-21 22:24:10 -07:00
peaceh-nv
e19309c98e
chore : Split more tests out of gpt tests (#3524) (#3674) 
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
 2025-04-21 20:19:40 -07:00
Yan Chunwei
e69d7bb1ad
unwaive multi-node test (#3721) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-22 07:56:29 +08:00
Tracin
81d1f4f1c0
Fix script options for engines. (#3622) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-21 18:59:18 +08:00
DylanChen-NV
8a8a55a2f0
fix: FP8 kv accuracy (#3675) 
* fix FP8 kv accuracy

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

* update doc

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

---------

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
 2025-04-21 15:59:31 +08:00
Tracin
8f17f3fa1d
Un-waive DS-V3-Lite tests. (#3621) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-21 14:54:24 +08:00
Yibin Li
b3ce6380cd
Fix: cherry-pick hmac encryption from main branch (#3635) 
* security fix cherry-pick changes from main

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>

* fix hmac in remote mpi session (#3649)

Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>
Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
 2025-04-21 13:08:03 +08:00