kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
520 Commits 25 Branches 59 Tags 2.1 GiB
03dad6287a
Commit Graph

520 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
nv-guomingz
03dad6287a
docs:fix https://nvbugs/5244616 by removing invalid links. (#3918) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-29 07:58:52 +08:00
Yan Chunwei
d244d7c891
waive a test (#3897) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-28 13:12:48 +08:00
Yiqing Yan
9f1305edf7
[Infra] Waive l0 test (#3895) 
* Waive L0 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

* Waive L0 tests

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>

---------

Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-28 11:00:39 +08:00
Yan Chunwei
c691477092
waive a test (#3886) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-27 16:47:18 +08:00
Kaiyu Xie
fde74feeb0
TRTLLM-4875 feat: Add version switcher to doc (#3871) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-04-26 07:18:49 +08:00
Enwei Zhu
29344e4697
[https://nvbugspro.nvidia.com/bug/5241495][fix] CUDA Graph padding with overlap scheduler (#3839) 
* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fuse

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-25 16:56:48 +08:00
Frank
7f8f227a13
fix: Add pre-download of checkpoint before benchmark. (#3772) 
* Add pre-download of checkpoint before benchmark.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Add missing remote code flag.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Move from_pretrained to throughput benchmark.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Move download and use snapshot_download.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Removed trusted flag.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Fix benchmark command in iteration log test.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

---------

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-04-25 09:02:53 +08:00
bhsueh_NV
ae910e9f00
fix: Fix/fused moe 0.19 (#3799) 
* fix bug of stream init

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix bug

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-24 22:25:41 +08:00
wili
33c4d49a77
[doc] Better document for Draft-Target-Model (DTM) speculative decoding (#3797) 
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-04-24 11:41:35 +08:00
Yukun He
ab2f663101
fix: Reduce memory usage in fused moe op associated with AutoTuning and fix moe fallback issue. (#3793) 
* Reduce memory usage in fused moe op associated with AutoTuning.
* Replace pre-defined bucket size strategy with a generating function based on the tune_max_num_tokens.
* Add free_memory logic of workspace in min_latency_mode fused moe path.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

* Fix fused_moe fallback issue. (#3652)

min_latency_mode is only set to False during warmup phase. Thus when it becomes true during inference, all tactics fall back to the default one and thus cause perf regression.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>

---------

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-24 10:14:26 +08:00
Yiqing Yan
e0691e6e27
Waive L0 tests (#3826) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-24 10:11:21 +08:00
Barry Kang
41b03714cd
Fix ModelOpt Mixtral AWQ OOM (#3714) (#3761) 
Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-24 03:17:48 +08:00
Fanrong Li
2f02263912
add know issue to deepseek doc. (#3800) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-23 17:21:50 +08:00
Ivy Zhang
f056d440ab
test: [nvbug: 5234494] skip_pre_ada for fp8 cases (#3718) 
* skip_pre_ada for fp8 cases

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* update

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* update after rebase

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

---------

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-23 16:54:40 +08:00
xinhe-nv
b0ac7c9ea9
test: skip failed cases on B200 (#3710) 
* add skip condition to tests

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

* fix error

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>

---------

Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-23 16:19:39 +08:00
Chuang Zhu
f08e59940b
skip disagg deepseek test if sm!=90 (#3720) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-04-23 14:47:45 +08:00
Ivy Zhang
3e56e402a5
remove benchmark test list (#3643) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-23 13:30:09 +08:00
Anurag Mukkara
851e2f5f9e
fix: PP4 fixes and cleanup (#3688) 
Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>
 2025-04-23 10:53:54 +08:00
Yukun He
a824946622
https://nvbugs/5141291: Fix convert.py script for Qwen model. (#3758) 
Include Qwen2VLDecoderLayer in the smooth_qwen2_model function.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-22 20:18:03 +08:00
Ivy Zhang
b11cb2fdd0
waive gemma on L20 (#3766) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-04-22 18:48:06 +08:00
dominicshanshan
792b71f412
Update ds v3 parameters in stress test. (#3676) 2025-04-22 18:04:17 +08:00
ruodil
793d0102d6
waive failed case in perf test, change default max_batch_size to 512 and write config.json to output log (#3656) 
Signed-off-by: Ruodi <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-22 14:53:21 +08:00
Yiqing Yan
52e6702725
Waive L0 test (#3756) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-22 14:26:19 +08:00
YueWeng
611ef8eb9d
test: Get Eagle tests working (#3593) (#3722) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
 2025-04-22 13:59:23 +08:00
nv-guomingz
ba15155a9c
doc:add torch examples link into torch backend documentation (#3749) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-21 22:24:10 -07:00
peaceh-nv
e19309c98e
chore : Split more tests out of gpt tests (#3524) (#3674) 
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
 2025-04-21 20:19:40 -07:00
Yan Chunwei
e69d7bb1ad
unwaive multi-node test (#3721) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-22 07:56:29 +08:00
Tracin
81d1f4f1c0
Fix script options for engines. (#3622) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-21 18:59:18 +08:00
DylanChen-NV
8a8a55a2f0
fix: FP8 kv accuracy (#3675) 
* fix FP8 kv accuracy

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

* update doc

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

---------

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
 2025-04-21 15:59:31 +08:00
Tracin
8f17f3fa1d
Un-waive DS-V3-Lite tests. (#3621) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-04-21 14:54:24 +08:00
Yibin Li
b3ce6380cd
Fix: cherry-pick hmac encryption from main branch (#3635) 
* security fix cherry-pick changes from main

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>

* fix hmac in remote mpi session (#3649)

Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>
Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
 2025-04-21 13:08:03 +08:00
liji-nv
a04b5851b9
fix: update test_user_buffers_mm_add_prologue atol (#3711) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-04-21 10:56:29 +08:00
QI JUN
fb8ddfaf86 tests: waive test_llm_multi_node (#3664) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-20 16:14:32 +00:00
Kaiyu Xie
422c1b30e9
doc: Update DeepSeek perf docs (#3693) 
* Update DeepSeek perf docs

Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>

* update

Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>

---------

Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-04-19 13:11:00 +08:00
nv-guomingz
c70b24c087
test:restore fp8 kv cache testing for L0 (#3671) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-19 12:28:35 +08:00
nv-guomingz
07688cd86f
test:sync waives.txt from main branch by disabling test_perf/gpt_350m-cppmanager case (#3685) 
Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>
 2025-04-19 01:22:38 +08:00
Zongfei Jing
1c6e85b133
Fix: nvbugs/5222698 variable not defined (#3630) 
* Fix: nvbugs/5222698 variable not defined

Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>

* Tidy code

Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>

---------

Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
 2025-04-18 21:41:20 +08:00
xiweny
5bf8fdca65
fix: nvbugs/5231298: pytorch allreduce issue (#3673) 
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
 2025-04-18 15:50:40 +08:00
Yanchao Lu
56c9dd42b7
infra: Add PR approval protection for the release branch (#3634) 
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-04-18 09:16:58 +08:00
Enwei Zhu
c8cea3001a
fix: FP8 quantized lm_head (NvBug 5214229) (#3567) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-17 15:18:34 +08:00
ruodil
b1a65c0e90
tests: change qa perf test to trtllm-bench (#3619) 
Signed-off-by: Ruodi <200874449+ruodil@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-04-17 13:58:38 +08:00
Tao Li @ NVIDIA
458203d805
update fp8 doc (#3647) 
Signed-off-by: taoli <litaotju@users.noreply.github.com>
Co-authored-by: taoli <litaotju@users.noreply.github.com>
 2025-04-17 13:16:07 +08:00
xiweny
5cc1d38958
fix: nvbugs/5187237: fix deterministic mode crash (#3448) 
* nvbugs/5187237 nvbugs/5112075: fix deterministic mode error


* remove waive
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>

* Revert "remove waive"

This reverts commit 0bf5486d19906d692bfb7a6262333c296b0087ac.

Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>

* revert ar fusion

Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>

---------

Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
 2025-04-17 12:01:57 +08:00
Enwei Zhu
e36092bd40
squash (#3642) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-17 10:41:56 +08:00
Ivy Zhang
715428cca9
test: add test cases for 0.19 release (#3608) 
* fix test name

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* add quickstart test for nemotron-ultra

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

* add rcca multi-node test case for deepseek-v3

Signed-off-by: Ivy Zhang <yanzh@nvidia.com>

* add rcca info

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

---------

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Ivy Zhang <yanzh@nvidia.com>
 2025-04-16 16:19:06 +08:00
Zhanrui Sun
3471d6ccf0
chore: bump version to 0.19.0 (#3598) 
Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-04-16 12:15:19 +08:00
narutolhy
ccd73c71a5
feat: Add stream generation task scaffolding examples (#3527) 
* stream generation task/controller

Signed-off-by: narutolhy <582909902@qq.com>

* edit README

Signed-off-by: narutolhy <582909902@qq.com>

* rename README

Signed-off-by: narutolhy <582909902@qq.com>

---------

Signed-off-by: narutolhy <582909902@qq.com>
 2025-04-16 11:33:55 +08:00
Yan Chunwei
409c294c4e
fix trtllm-bench mgmn (#3563) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-16 11:04:09 +08:00
Yan Chunwei
63f3fba679
waive test_llm_multi_node_pytorch (#3592) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-16 10:49:07 +08:00
Enwei Zhu
44da0e8d60
fix: LLM API _hf_model_dir for non-cached case (#3562) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-16 10:39:34 +08:00