kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-30 07:33:48 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
3,059 Commits 28 Branches 64 Tags 2 GiB
f4e1cc7b39
Commit Graph

659 Commits

Author SHA1 Message Date
Chang Liu
e5b6d335eb
[https://nvbugs/5568961][fix] Fix a merge conflict (cherrypick from PR 8365) (#8553) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-10-23 14:05:16 +08:00
Lizhi Zhou
3f82cdbdad
[https://nvbugs/5582277][fix] rework DisaggPPTerminationHandler to fix hang issue (#8519) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-10-23 09:43:59 +08:00
Bo Deng
9e30f14da8
[https://nvbugs/5565549][fix] unwaive test_disaggregated_spec_dec_bat… (#8500) 
Signed-off-by: Bo Deng <deemod@nvidia.com>
 2025-10-22 14:59:59 +08:00
Ivy Zhang
f904348cd6
[TRTLLM-8580][test] save runtime report periodically (#8312) (#8455) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-10-20 10:54:24 +08:00
Yukun He
437a3fc642
[None][chore] Remove duplicate log outputs in test_perf.py (#8418) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-10-17 14:11:32 +08:00
ruodil
20c2de4924
[None][test] cherry-pick: add test-model-suites in integration conftest.py (#8388) 
Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com>
Co-authored-by: Ruodi Lu <ruodil@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-10-15 23:26:32 -07:00
Patrice Castonguay
7862372ee2
[https://nvbugs/5552889][fix] fix: Prevent empty batch when using attention DP with disagg (#8372) 
Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
 2025-10-16 09:11:04 +08:00
Stanley Sun
cce97e6e15
[TRTLLM-8113][test] Add pytorch workflow e2e tests with pp enabled (#8357) 
Signed-off-by: Stanley Sun <stsun@nvidia.com>
 2025-10-15 15:09:21 +08:00
xiweny
d5b79268e7
[https://nvbugs/5565565] [fix] fp8 wideep support sm103 (#8228) 
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
 2025-10-15 10:17:08 +08:00
bhsueh_NV
66aa88739b
[https://nvbugs/5574556][fix] fix bug of Qwen3_235B_A22B::test_fp8 CI (#8351) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-10-14 15:26:15 +08:00
Lizhi Zhou
553ff3402a
[https://nvbugs/5550671][fix] fix disagg-serving multinodes test failure (#8307) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-10-14 08:01:00 +02:00
Chuang Zhu
6a73f079fe
[https://nvbugs/5465642][fix] Increase server timeout to wait weight loading (#8297) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-10-14 07:55:31 +02:00
Enwei Zhu
598e88594c
[https://nvbugs/5568951][fix] Fix guided decoding disagg tests (#8311) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-10-13 18:55:28 +08:00
Chuang Zhu
ad0e91a174
[https://nvbugs/5546202][fix] Fix concurrent bug for NIXL cache transceiver (#8147) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-10-13 09:40:56 +02:00
Ivy Zhang
6a42a9649b
[None][chore] Update test configs for release (#8224) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-10-13 14:07:33 +08:00
Ivy Zhang
bca5e29387
[None][chore] Update constaintfor release (#8211) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-10-13 11:14:24 +08:00
Yukun He
1ca84e1a25
[https://nvbugs/5536131][fix] Fix illegal access issue when scale is not provided in Llama3/4. (#7960) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-10-07 23:47:00 -07:00
Enwei Zhu
d650320de4
[None][infra] Improve the failure message for accuracy test suite (#7994) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-09-26 10:04:47 +08:00
Guoming Zhang
202bed4574 [None][chroe] Rename TensorRT-LLM to TensorRT LLM for source code. (#7851) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-25 21:02:35 +08:00
Guoming Zhang
9f0f52249e [None][doc] Rename TensorRT-LLM to TensorRT LLM for homepage and the … (#7850) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-25 21:02:35 +08:00
Chuang Zhu
791e73edf6
[https://nvbugs/5536141][fix] fix_disagg_single_gpu_test (#7990) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-09-25 02:07:22 -07:00
Mike Iovine
42c2ec3239
[https://nvbugs/5473781][fix] Fix llama 4 FP8 for PP>1 (#7220) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-09-24 12:16:27 -04:00
Pamela Peng
b1dc84b4a3
[TRTLLM-7399][test] Add DS-R1/Qwen3 test cases for RTX 6000 (#7662) 
Signed-off-by: Pamela <179191831+pamelap-nvidia@users.noreply.github.com>
Signed-off-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-09-24 11:40:26 -04:00
Enwei Zhu
a1a57e83b8
[TRTLLM-5235][feat] Enable regex and EBNF grammar in trtllm-serve (#7925) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-09-24 18:30:23 +08:00
xinhe-nv
b8bfa63197
[None][chore] add test_w4_1gpu[True-True-cutlass-fp8] & TestKimiK2::test_fp8_blocks… (#7944) 
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-09-24 03:25:17 -07:00
yufeiwu-nv
f323b74d42
[None][test] Update llm_models_root to improve path handling on BareMetal environment (#7876) 
Signed-off-by: yufeiwu <230315618+yufeiwu-nv@users.noreply.github.com>
Signed-off-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com>
 2025-09-24 17:35:57 +08:00
xinhe-nv
62563760fb
[None][chore] update chunked prefill cases (#7921) 
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-09-24 15:14:49 +08:00
Yueh-Ting (eop) Chen
cf100933cc
[TRTLLM-6341][feature] Support SWA KV cache reuse (#6768) 
This merge request attempts to support more SWA KV cache functionality
inside the KV cache manager. Before this merge request, the KV cache for
sliding window attention (SWA) only holds "window size" number of blocks
and reuse them in a cyclic manner. We will not be able to utilize more
GPU memory with this design, leading to a limited max batch size
throughput. Additionally, we will not be able to support KV cache reuse
with this design.

In this MR, we change such behavior to let the manager write blocks in
a linear manner. With a linear block writing behavior, as the attention
window moves on, the out-of-window (OOW) blocks will be detached. Right
now for the sake of a correct feature first, we directly offload the
OOW block from the primary block pool (GPU memory) to the secondary
block pool (host memory). We will improve this in the future by
delegating the block movement to the eviction policy.

KV cache reuse for SWA is not developed in this merge request and will
be amended in a follow-up merge request.

Writing the blocks linearly, the maximum number of blocks allocated for
a sequence(`GenerationRequest`) is the "max sequence length" specified.
The `GenerationRequest` that stores the cache block bookkeeping
structure will now keep "max sequence length" tokens of blocks.

Given the above, main changes are (more context in the MR):
- Remove "cyclic" concept under the kv cache manager, such concept
  originally guards the block reuse under kv cache manager.
- Add detach mechanism and have it under `KVCacheManager::addToken`.
  Please note that detach is still guarded off for SWA when reuse
  is enabled. A follow-up merge request will proceed to improve this.
- Enforce "max sequence length" to be a non-optional parameter to
  the `KVCacheManager`/`BlockManager`
- Let all window size resource pool get identical proportion of memory
- Fix free memory calculation under `resource_manager.py`

Signed-off-by: eopXD <yuehtingc@nvidia.com>
Co-authored-by: Tomer Asida <tasida@nvidia.com>
 2025-09-24 14:28:24 +08:00
Lizhi Zhou
7550251988
[TRTLLM-7182][test] add multi-nodes test for disagg-serving (#7470) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-09-24 08:31:56 +08:00
Zheng Duan
e3c1a9409f
[TRTLLM-6549][fix] add kv cache time output back (#7798) 
Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>
 2025-09-23 14:12:42 -04:00
Pengbo Wang
a4b4ed4535
[None][fix] Fix and add test for TRTLLM MoE backend (#7755) 
Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>
 2025-09-23 11:26:25 +08:00
yunruis
126cd707e3
[None][opt] Add batch waiting when scheduling (#7416) 
Signed-off-by: yunruis <205571022+yunruis@users.noreply.github.com>
Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com>
 2025-09-23 10:27:37 +08:00
xinhe-nv
9c1b75e978
[TRTLLM-7070][feat] add gpt-oss chunked prefill tests (#7779) 
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-09-22 00:12:43 -07:00
Wanli Jiang
f5bfd68a50 [https://nvbugs/5509024][fix] Print full parsed outputs and update keywords for multimodal model (#7670) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-22 14:28:38 +08:00
Yi Zhang
f9c9c3f50a [https://nvbugs/5355219][fix] Fix trtllm moe backend test config and Qwen3 MoE multi node (#7724) 
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-22 14:28:38 +08:00
Ivy Zhang
022bc96fb6 [https://nvbugs/5512734][fix] Update kv cache config for maverick (#7710) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-22 14:28:38 +08:00
Yuxian Qiu
2d46dda6a7 [https://nvbugs/5448754][fix] Download HF model for all nodes. (#6824) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-22 14:28:38 +08:00
Lizhi Zhou
293d9fb612 [https://nvbugs/5448767][fix] disable kv cache reuse for disagg pp>1 tests (#7354) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-22 14:28:38 +08:00
Yuxian Qiu
7d28acdbf0
[https://nvbugs/5522332][fix] Pin numpy version for Gemma. (cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/7783) (#7797) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-09-19 18:50:40 +08:00
xinhe-nv
efb763402f
[None][chore] Add failed cases into waives.txt (#7841) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-09-19 17:59:47 +08:00
Ivy Zhang
6b33bcced2
[None][test] Add accuracy benchmark in stress test (#7561) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
 2025-09-19 16:09:46 +08:00
ruodil
c5453103d6
[None][test] add deepseek r1/v3 model with chunked prefill cases (#7124) 
Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com>
 2025-09-19 11:12:53 +08:00
fredricz-20070104
fc4e6d3702
[TRTLLM-7183][test] Feature fix model issue for disagg serving (#7785) 
Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com>
 2025-09-19 10:12:55 +08:00
Yuxian Qiu
d6ebcf7c4a
[TRTLLM-6994][feat] FP8 Context MLA integration (Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6059 from release/1.1.0rc2) (#7610) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-09-19 09:40:49 +08:00
QI JUN
7646da2d85
[None][ci] set TORCHINDUCTOR_COMPILE_THREADS correctly (#7800) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-09-19 07:19:50 +08:00
sunnyqgg
80dd8fe197
[TRTLLM-6746][feat] Enable two-model spec dec for MTP Eagle (#7001) 
Signed-off-by: qgai <qgai@nvidia.com>
 2025-09-18 12:05:36 -04:00
xinhe-nv
d3a907131a
[https://nvbugs/5519462][fix] Add failed cases into waives.txt (#7817) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com>
 2025-09-18 20:01:06 +08:00
Wanli Jiang
fe104dc20d
[TRTLLM-7918][feat] Support kvcache reuse and chunk prefill for phi4mm (#7723) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-09-18 17:37:16 +08:00
Wanli Jiang
a7ca0fff54
[TRTLLM-6577][feat] Support nano_v2_vlm in pytorch backend (#7207) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-09-18 16:26:20 +08:00
dongfengy
2ae08bd1b8
[https://nvbugs/5519530][fix] Fix gptoss 2-gpu test (#7819) 
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
 2025-09-18 16:01:53 +08:00