kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,237 Commits 25 Branches 59 Tags 2.1 GiB
947571c311
Commit Graph

1237 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Chuang Zhu
947571c311
Fix buffer count (#5007) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-06-09 14:01:13 +08:00
Yan Chunwei
f4bfb8e49d
ci: unwaive llmapi launch test (#4991) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-06-09 13:25:43 +08:00
Daniel Stokes
3a4851b7c3
feat: Add Mixture of Experts FP8xMXFP4 support (#4750) 
Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>
 2025-06-09 13:25:04 +08:00
amitz-nv
77e8d739f1
[TRTLLM-4987][feat] Support generation logits in TRTLLMSampler (#4819) 2025-06-09 06:30:01 +03:00
Yechan Kim
8b4104d34a
feat: add HyperCLOVAX-SEED-Vision support in refactored way (#4799) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-06-09 11:04:04 +08:00
Julien Demouth
bb79ba7c35
Edits for tech blog 4 (#5006) 
Signed-off-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
Co-authored-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
 2025-06-09 09:38:41 +08:00
nv-guomingz
78472339b3
fix:https://nvbugs/5324252 (#4925) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-06-09 01:15:45 +08:00
Omer Ullman Argov
8731f5f14f
chore: Mass integration of release/0.20 (#4898) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Hui Gao <huig@nvidia.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Signed-off-by: Ruodi <200874449+ruodil@users.noreply.github.com>
Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com>
Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>
Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
Signed-off-by: moraxu <mguzek@nvidia.com>
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Co-authored-by: Yiqing Yan <yiqingy@nvidia.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Co-authored-by: HuiGao-NV <huig@nvidia.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: Bo Li <22713281+bobboli@users.noreply.github.com>
Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com>
Co-authored-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com>
Co-authored-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>
Co-authored-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>
Co-authored-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Co-authored-by: Faraz <58580514+farazkh80@users.noreply.github.com>
Co-authored-by: Michal Guzek <moraxu@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
 2025-06-08 23:26:26 +08:00
Mike Iovine
ec0d984656
[nvbug/5280806][fix] Fix 2 model spec decode flow (#4807) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-06-08 07:40:02 -04:00
Yanchao Lu
9e05613679
[Infra] - Update JNLP container config (#5008) 
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-06-08 16:44:09 +08:00
nv-guomingz
786e32d56f
chore:update modelopt to 0.31 (#5003) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-06-08 15:55:33 +08:00
dongxuy04
1e369658f1
feat: large-scale EP(part 6: Online EP load balancer integration for GB200 nvfp4) (#4818) 
Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
Signed-off-by: ShiXiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>
Co-authored-by: ShiXiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>
 2025-06-08 10:25:18 +08:00
QI JUN
5ee0de7f2a
Resubmit #4894 (#4969) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-08 04:42:15 +08:00
nv-guomingz
d8abb91dc8
chore:set the flashinfer to 0.2.5. (#5004) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-06-07 20:42:09 +08:00
Bo Li
f414a079ad
chore: Change the type annotations of input_ids and position_ids to int32. (#4632) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-06-07 16:10:47 +08:00
Ivy Zhang
7dce328ad6
[TRTLLM-5692][tests] Add speculative decoding test cases on torch flow (#4940) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Ruodi Lu <ruodil@nvidia.com>
Co-authored-by: Ruodi Lu <ruodil@nvidia.com>
 2025-06-07 11:18:32 +08:00
nv-guomingz
0c7dd660d8
fix:https://nvbugs/5324248 (#4973) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-06-07 04:14:07 +08:00
Jinyang Yuan
20d0649f19
[feat] Support XQA-based MLA on SM120 (#4858) 
Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
Co-authored-by: Yao Yao <lowsfer@users.noreply.github.com>
Co-authored-by: peaceh-nv <103117813+peaceh-nv@users.noreply.github.com>
 2025-06-06 22:32:49 +08:00
Fanrong Li
75d020cf07
fix: fix cuda graph padding for spec decoding (#4853) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-06-06 22:21:42 +08:00
Anthony Chang
eeb555e37b
chore: memoize weight shuffle index to speed up weight preproc in moe_backend=TRTLLM (#4826) 
Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>
 2025-06-06 16:13:54 +08:00
QI JUN
1b963c17c0
CI: waive test_llm_multi_node_with_postproc (#4977) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-06 14:19:56 +08:00
xinhe-nv
564472168e
test: [CI] Add failed cases into waives.txt (#4966) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-06-06 10:30:15 +08:00
juney-nvidia
a761cc2f8d
doc: refinement based on Julien's feedbacks (#4967) 
Signed-off-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
 2025-06-06 08:56:14 +08:00
QI JUN
ec50684d80
Revert "fix a bug of global cuda graph dummy request" (#4970) 2025-06-06 08:54:45 +08:00
juney-nvidia
37ac564190
doc: expose Large-scale EP design and implementation tech blog in the main… (#4960) 
Signed-off-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
 2025-06-05 22:51:25 +08:00
Yiteng Niu
d2c311c9d3
infra: update jnlp version in container image (#4944) 2025-06-05 22:36:10 +08:00
Kaiyu Xie
5a5427f86e
blog: Scaling Expert Parallelism in TensorRT-LLM (Part 1: Design and Implementation of Large-scale EP) (#4958) 
Signed-off-by: juney-nvidia <143764042+juney-nvidia@users.noreply.github.com>
Co-authored-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
Co-authored-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
Co-authored-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
 2025-06-05 22:24:04 +08:00
qsang-nv
180b91f957
update fmha_v2 (#4895) 
Signed-off-by: Qidi Sang <200703406+qsang-nv@users.noreply.github.com>
 2025-06-05 22:14:28 +08:00
dongjiyingdjy
51652b9b2b
feat : add PositionEmbeddingType=0 to xqa support (#4934) 
Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
 2025-06-05 21:50:42 +08:00
QI JUN
bfa877a22e
Fix: fix autodeploy (#4957) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-05 21:06:55 +08:00
QI JUN
154f7cc40a
fix a bug of global cuda graph dummy request (#4894) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-05 19:47:40 +08:00
Yiqing Yan
7e921c78b5
Waive L0 tests (#4953) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-06-05 19:36:48 +08:00
Shunkangz
3eae58ca36
Add disaggregated unittest (#4899) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-06-05 19:14:31 +08:00
ixlmar
a1526356aa
[TRTLLM-5630] restore free_gpu_memory_fraction=0.9 in tests (#4859) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-06-05 10:46:29 +01:00
QI JUN
b8c5e3892b
Revert "fix: build_config in TorchLlmArgs and avoid invalid args" (#4949) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-05 17:43:30 +08:00
QI JUN
d5a8079eb6
Revert "[infra] Unwaive unittests/_torch" (#4950) 2025-06-05 17:21:07 +08:00
Lucas Liebenwein
743fb0a159
[AutoDeploy] _AutoDeployLlmArgs as primary config object (#4891) 
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
 2025-06-05 17:20:55 +08:00
QI JUN
91e8d43d66
CI: waive test_llm_get_queued_stats (#4945) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-05 16:44:56 +08:00
ixlmar
6437756da8
fix: handle OOMs during KV cache estimation (#4690) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-06-05 10:02:26 +02:00
xinhe-nv
1c3091c63b
tests: [TRTQA-2906] add benchmark serving tests (#4901) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-06-05 14:33:03 +08:00
Netanel Haber
ddbaa5ef80
Only pass fast_build=true to non-pytorch backend (#4920) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
 2025-06-05 13:30:17 +08:00
Yiqing Yan
9ceef983c0
Waive L0 tests (#4927) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-06-05 11:09:01 +08:00
xinhe-nv
50a74a1daa
tests: fix 5273697 (#4685) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-06-05 10:39:21 +08:00
Shiyu Li
b0d287c9b7
[TRTLLM-4647][fix] Fix the no fusion allreduce hanging (#4594) 
Signed-off-by: Shiyu Li <shili@nvidia.com>
 2025-06-04 18:26:13 -07:00
Mike Iovine
8433091630
[infra] Unwaive unittests/_torch (#4919) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-06-05 08:49:37 +08:00
Lucas Liebenwein
f9d45e03a4
[AutoDeploy] deprecate CI post-merge tests and keep them for local testing (#4892) 
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
 2025-06-05 08:27:17 +08:00
Yan Chunwei
8e0d96fcc6
fix: LLM invalid arg in a test (#4922) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-06-05 08:00:32 +08:00
Yuxian Qiu
6b3242654e
fix: Fix broken vanilla moe since FusedMoE refactor. (#4897) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-06-05 03:56:41 +08:00
Yi Zhang
1fca654bfd
tests: Update gb200 test case (#4754) 
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
 2025-06-04 18:49:20 +08:00
ixlmar
2bbb6b5976
chore: introduce KvCacheCreator (#4581) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-06-04 11:03:17 +02:00