kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
2,432 Commits 25 Branches 59 Tags 2.1 GiB
6f245ec78b
Commit Graph

2432 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dominicshanshan
6f245ec78b
[None][chore] Mass integration of release/1.0 (#6864) 
Signed-off-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Bo Deng <deemod@nvidia.com>
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
Signed-off-by: qqiao <qqiao@nvidia.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: raayandhar <rdhar@nvidia.com>
Co-authored-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com>
Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com>
Co-authored-by: Yiqing Yan <yiqingy@nvidia.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Bo Deng <deemod@nvidia.com>
Co-authored-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
Co-authored-by: Emma Qiao <qqiao@nvidia.com>
Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Co-authored-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com>
Co-authored-by: Raayan Dhar <58057652+raayandhar@users.noreply.github.com>
Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-08-22 09:25:15 +08:00
Daniel Stokes
f7c597ec40
[None][perf] Make finalize fusion part of the tactic selection logic (#6915) 
Signed-off-by: djns99 <40156487+djns99@users.noreply.github.com>
 2025-08-21 14:08:03 -07:00
Fridah-nv
e18dacc931
[#4403][refactor] Move fusion, kvcache, and compile to modular inference optimizer (#7057) 
Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
 2025-08-21 10:30:36 -07:00
Emma Qiao
344bc4575d
[None][infra] Waive failed case for main branch (#7129) 
Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-08-22 00:08:55 +08:00
Dimitrios Bariamis
f49dafe0da
[https://nvbugs/5394409][feat] Support Mistral Small 3.1 multimodal in Triton Backend (#6714) 
Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
Signed-off-by: Dimitrios Bariamis <dbari@users.noreply.github.com>
Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
 2025-08-21 18:08:38 +02:00
brb-nv
9a2b44d0f2
[None][chore] No-op changes to support context parallelism in disaggregated serving later (#7063) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-08-21 08:21:27 -07:00
Yuan Tong
90bfc8cc29
[https://nvbugs/5453827][fix] Fix RPATH of th_common shared library to find pip-installed NCCL (#6984) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-08-21 17:58:30 +08:00
ChristinaZ
c7269ea93a
[https://nvbugs/5392414] [fix] Add customized default routing method (#6818) 
Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>
 2025-08-21 16:58:41 +08:00
Farshad Ghodsian
2d40e8750b
[None][doc] Update gpt-oss deployment guide to latest release image (#7101) 
Signed-off-by: Farshad Ghodsian <47931571+farshadghodsian@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-08-21 02:33:07 -04:00
bhsueh_NV
ba0a86e0bb
[https://nvbugs/5437405][fix] qwen3 235b eagle3 ci (#7000) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-08-21 01:17:32 -04:00
Fridah-nv
647a52698a
[https://nvbugs/5443039][fix] Fix AutoDeploy pattern matcher for torch 2.8 (#7076) 
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
 2025-08-21 01:14:51 -04:00
Yao Yao
cbcea33279
[fix]: use safeInitRowMax instead of fp32_lowest to avoid NaN (#7087) 
Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
 2025-08-20 22:12:21 -07:00
xinhe-nv
21f4434404
[None][chore] waive failed cases on H100 (#7084) 
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-08-21 11:15:23 +08:00
Fan - Yunfan
41ff4901ee
[None][fix] Fix const modifier inconsistency in log function declaration/implementation (#6679) 
Signed-off-by: fanyunfan <2569548856@qq.com>
Co-authored-by: fanyunfan <2569658856@qq.com>
Co-authored-by: Yunfan Fan <46273019+fyf2016@users.noreply.github.com>
 2025-08-21 11:08:11 +08:00
Fridah-nv
f03053b4dd
[None][fix] update accelerate dependency to 1.7+ for AutoDeploy (#7077) 
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
 2025-08-20 19:52:37 -07:00
BatshevaBlack
9f51f8d20c
[None][infra] Upgrade UCX to v1.19.x and NIXL to 0.5.0 (#7024) 
Signed-off-by: Batsheva Black <132911331+BatshevaBlack@users.noreply.github.com>
Signed-off-by: Bo Deng <deemod@nvidia.com>
Co-authored-by: Bo Deng <deemod@nvidia.com>
 2025-08-20 22:49:55 -04:00
Chang Liu
75b8a90816
[None][fix] Fix llama4 multimodal by skipping request validation (#6957) 
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
 2025-08-20 21:58:53 -04:00
Yechan Kim
0893afae3d
[TRTLLM-6771][feat] Support MMMU for multimodal models (#6828) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-21 08:54:12 +08:00
bhsueh_NV
73d2daa386
[https://nvbugs/5457489][fix] unwaive some tests (#6991) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-08-21 08:49:57 +08:00
QI JUN
a918de710a
[None][ci] move some tests of b200 to post merge (#7093) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-08-20 19:43:40 -04:00
Jin Li
e5e417019b
[None][chore] Only check the bindings lib for current build (#7026) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-08-20 14:17:17 -04:00
Dom Brown
92daec1115
[TRTLLM-7348] [feat] Enable Cross-Attention to use XQA kernels for Whisper (#7035) 
Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-08-20 10:11:25 -04:00
Yuhao Yao
8ac7dec623
[None][fix] Fix W4A8 MoE kernel issue (#7072) 
Signed-off-by: yuhyao <827623970@qq.com>
 2025-08-20 06:52:47 -04:00
Emma Qiao
f84dd64250
[None][infra] Waive failed tests on main branch 8/20 (#7092) 
Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-08-20 06:33:44 -04:00
Robin Kobus
b95cab2a7c
[None][ci] move unittests to sub-directories (#6635) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-08-20 05:42:22 -04:00
Kanghwan
983fb8e607
[None][chore] Update namelist in blossom-ci (#7015) 
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
 2025-08-20 09:29:01 +02:00
Zhenhuan Chen
20f54cb272
[None][fix] fix scaffolding dynasor test (#7070) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-08-20 15:20:46 +08:00
Yueh-Ting (eop) Chen
020fed97b6
[TRTLLM-6341][chore] Preliminary refactors on the kv cache manager before supporting swa kv cache reuse (#6767) 
This MR is a preliminary MR for implementing the SWA reuse mechanism for
the kv cache manager. Please be aware that **no functional change is
intended** in this merge request. The purpose of the clean-up is to
decouple and remove existing functions for the up-coming SWA KV cache
reuse change to be more natural and easier to review.

Right now, (1) streamLLM, and (2) beam search with SWA, are broken. We
do not want to complicate the code base by stacking more features upon
something that does not work. This MR prunes out the logic and add
assertions so we can come back and re-support the broken feature and
remove the assertion.

Since streamLLM (sink attention) is broken now, assertion is added
under `KVCacheManager` ctor to guard for the value of
`mSinkBlockTokenLength` and `mSinkBubbleLength`. Compute logics relate
to it are pruned.

The beam search with SWA will still be broke when introducing the SWA
KV cache reuse. We will revisit this problem in the future.

On top of this, we should make an effort to update the [supporting matrix](https://github.com/NVIDIA/TensorRT-LLM/blob/feat/1.0_doc_dev/docs/source/1.0/features/feature-combination-matrix.md)
of the kv cache manager after merging the support of SWA KV cache reuse.

Changes are listed as following:
- Separate `KVCacheManager::updateToken` into `KVCacheManager::addToken`
  and `KVCacheManager::removeToken`. The functionality should be
  decoupled.
- Push utility `cacheSequenceBlockOffsets` and `cacheNewBlockOffset` from
  `KVCacheManager` down to `WindowBlockManager`. `KVCacheManager`-exposed
  functions should be real utilities that users of the structure can
  leverage. Implementation-detailed function calls should not exist at
  this level.
- Simplify "is shared last context block" logic under
  `KVCacheManager::addSequence`.

Since no functional change is intended in this merge request, no test
case is added. Several comments are added for future test coverage
reminder.

For `LlmRequestTest.ParamTest`, `streaming=True` is commented out
because we guard sink attention with assertion now.

In `capacitySchedulerTest`, `addToken` action to `crossKVCacheManager`
is removed because in encoder-decoder model, generation tokens are
added only to the decoder and not to the encoder.

Signed-off-by: eopXD <yuehtingc@nvidia.com>
 2025-08-20 13:57:57 +08:00
Iman Tabrizian
e27088421e
[None][infra] "[TRTLLM-6960][fix] enable scaled_mm tests (#6936)" (#7059) 
Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>
 2025-08-20 01:45:09 -04:00
xinhe-nv
9e71b4fda4
[TRTLLM-7205][feat] add llama4 tp4 tests (#6989) 
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-08-20 13:22:05 +08:00
Leslie Fang
3f6a9267f1
[None][infra] update feature_combination_matrix of disaggregated and chunked prefill (#6661) 
Signed-off-by: leslie-fang25 <leslief@nvidia.com>
 2025-08-20 13:14:34 +08:00
Chang Liu
ce53832610
[TRTLLM-7326][feat] Add standalone multimodal encoder (#6743) 
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
 2025-08-19 21:42:50 -07:00
Ivy Zhang
fc85e3db1c
[None][fix] fix llmapi import error (#7030) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-08-19 22:58:13 -04:00
Bo Deng
30da5d3cc4
[None][chore] unwaive test_disaggregated_genbs1 (#6944) 
Signed-off-by: Bo Deng <deemod@nvidia.com>
 2025-08-20 09:57:35 +08:00
Fridah-nv
c02592d051
[None][autodeploy] Add group attention pattern for solar-pro-preview (#7054) 
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
 2025-08-19 18:57:09 -04:00
Jinyang Yuan
0e30fe4372
[None][fix] Fix assertion errors of quantization when using online EPLB (#6922) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-08-19 11:28:36 -07:00
Michal Guzek
7334f9390c
[None][fix] Accommodate Phi3/4 to work with ModelOpt's FP8 ckpts in Torch (#6761) 
Signed-off-by: Michal Guzek <mguzek@nvidia.com>
 2025-08-19 09:22:46 -07:00
Yanchao Lu
d26a5a93ad
[https://nvbugs/5451296][bug] Cherry-pick #7017 from release/1.0 branch (#7043) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
 2025-08-19 11:25:05 -04:00
pcastonguay
e07fcc3a22
[https://nvbugs/5444937][chore] Fixing KV events tests (#7004) 
Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
 2025-08-19 11:18:04 -04:00
zhhuang-nv
7e135d2ea7
[None][feat] Use Separate QKV Input Layout for Context MLA (#6538) 
Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>
 2025-08-19 22:04:48 +08:00
Emma Qiao
8f95f35503
[None][infra] Waive failed tests on main (#7037) 
Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-08-19 09:31:07 -04:00
Yiqing Yan
07506bccbe
[None][chore] Remove duplicate test waives (#7044) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-08-19 21:04:31 +08:00
Fanrong Li
655d0f48d0
[https://nvbugs/5455140][fix] unwaive DSR1-fp4 throughput_tp8 (#7022) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-08-19 20:48:05 +08:00
tomeras91
f0bfb49219
[https://nvbugs/5458874][fix] Fix Nemotron-H flaky CUDA graph / overlap scheduler test (#6996) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
 2025-08-19 15:45:06 +03:00
amitz-nv
a54c53652b
[TRTLLM-7263][fix] Prevent recreation of cublas handles in lora_grouped_gemm every call (#6968) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-08-19 15:39:56 +03:00
Xianjie Qiao
19667304b5
[None] [chore] Update wide-ep genonly scripts (#6995) 
Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-08-19 07:44:07 -04:00
Kaiyu Xie
9a74ee9dae
[None] [doc] Add more documents for large scale EP (#7029) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-08-19 19:04:39 +08:00
Zero Zeng
953f4fd69e
[None][fix] acceptance rate calculation fix in benchmark_serving (#6746) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-19 17:29:36 +08:00
xinhe-nv
2c86cee38c
[None][chore] Remove closed bugs (#6969) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-08-19 16:01:33 +08:00
Shunkangz
54ec2c1af1
[None][opt] Add batch wait timeout in fetching requests (#6923) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-08-19 03:50:08 -04:00