kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
4,155 Commits 25 Branches 59 Tags 2.1 GiB
f6b0ddd61d
Commit Graph

4155 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Erin
c75d7cd684
move BuildConfig functional args to llmargs (#3036) 
Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
 2025-03-29 02:20:18 +08:00
Robin Kobus
3ee4332fb1
refactor: Replace DecoderFinishedEvent with CudaEvent in decoder classes (#3078) 
- Updated the `forwardAsync` method in `GptDecoderBatched` and `iGptDecoderBatched` to return `CudaEvent` instead of `DecoderFinishedEventPtr`, simplifying event handling.
- Removed the `DecoderFinishedEvent` class and its associated usage across various files, streamlining the codebase.
- Adjusted related methods and Python bindings to accommodate the new event structure, ensuring compatibility and maintaining functionality.

These changes enhance the clarity and efficiency of the decoding process in the batch manager.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-03-28 14:50:52 +08:00
Robin Kobus
45134d7095
refactor: Improve decoder finalize function (#3077) 
* refactor: Update gatherTree function to accept CUDA stream parameter

This commit modifies the gatherTree function signature to include a runtime::CudaStream parameter, enhancing flexibility in stream management. Additionally, it removes unnecessary buffer manager parameters and stream handling from the function, streamlining the code. The finalize method in GptDecoderBatched is also updated to reflect these changes, improving clarity and maintainability in the decoding process.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: Update GptDecoderBatched finalize

This commit refactors the GptDecoderBatched class to improve method signatures and reduce code complexity:

- Modified finalize method to accept DecoderState as a parameter
- Updated method signatures to work with the new DecoderState approach
- Improved code organization and readability

The changes continue the ongoing refactoring to centralize decoder state management and simplify the decoder implementation.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

---------

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-03-28 14:33:59 +08:00
BatshevaBlack
3e37531c6a
feat: Add BW measurement (#3070) 2025-03-28 10:53:00 +08:00
Aurelien Chartier
3de82c41cd
Pytorch PP + attention DP support (#3044) 
Signed-off-by: Aurelien Chartier <achartier@nvidia.com>
 2025-03-28 00:11:19 +08:00
Fanrong Li
ec03159e60
fix: Waive twoshot to fix acc issue (#3066) 
* waive twoshot to fix acc issue

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>

---------

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-03-27 21:38:52 +08:00
Fanrong Li
644a01cbbe
test: Add gpqa tests for DeepSeek models (#3063) 
* Add gpqa accuracy test script
* Add gpqa accuracy tests
* Update DeepSeek-v3 doc
* Update qa test list

---------

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-03-27 19:47:06 +08:00
Yan Chunwei
87ab794aa2
fix: fix hang in mgmn with trtllm-llmapi-launch command (#3119) 
* init

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* restore

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-03-27 18:45:43 +08:00
Fanrong Li
0976360204
add support for MTP+cuda_graph_padding. (#3096) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-03-27 16:06:14 +08:00
xiweny
6979afa6f2
test: reorganize tests folder hierarchy (#2996) 
1. move TRT path tests to 'trt' folder
2. optimize some import usage
 2025-03-27 12:07:53 +08:00
Yan Chunwei
82edd90350
fix gpus_per_node in trtllm-bench when world_size < device_count (#3007) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-03-27 09:31:40 +08:00
Dom Brown
60d4dacc47
Port multi GPU changes to GitHub (#3027) 
Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-03-27 05:55:03 +08:00
Suyog Gupta
047f2b234d
perf: [AutoDeploy] Enable AutoDeploy as a backend in trtllm-bench (#3041) 
* Enable AutoDeploy as a backend in trtllm-bench

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* update how caches are resized

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* fix: files permission from 100755 to 100644

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* some comments

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* lint

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* lint

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* lint

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* lint

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* Fix function name

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* refactor

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* Remove spurious change

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* Add cursor generated doc strings

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* re-enable ad test

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* some perf cleanup

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* debug ci

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* ensure that overlap scheduler is enabled

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

* Reorder the tests

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>

---------

Signed-off-by: Suyog Gupta <suyogg@nvidia.com>
Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-03-26 14:33:14 -07:00
wili
3e035f2219
v1.2 (#3082) 
Signed-off-by: wili <wili@nvidia.com>
 2025-03-26 23:31:29 +08:00
Robin Kobus
d9522c5906
feat: Update cutlass (#2981) 
* chore: update cutlass to v3.8.0

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: update include directives for consistency and organization in weightOnlyBatchedGemv headers

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* Fix fpA_intB_gemm compilation

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

---------

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-03-26 22:36:27 +08:00
Jinyang Yuan
6b583f6f83
perf: Enable CUDA graphs when attention DP is used and active requests on different GPUs are uneven (#3010) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
Co-authored-by: raccoonliukai <raccoonliu@tencent.com>
 2025-03-26 21:09:25 +08:00
Robin Kobus
3c3629c52a
refactor: simplify forward methods in GptDecoderBatched (#3076) 
* refactor: Remove ForwardType enum from GptDecoderBatched

 - Remove ForwardType enum from GptDecoderBatched
 - Simplify forwardDispatch and forwardDecoder methods

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: Remove forwardDecoder method from GptDecoderBatched

- Eliminate the forwardDecoder method to streamline the decoding process.
- Update forwardDispatch to directly call forwardAsync when input batch size is greater than zero.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

* refactor: Move event handling from forwardDispatch to forwardAsync

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>

---------

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-03-26 20:45:04 +08:00
Robin Kobus
94dd456bd0
refactor: Remove speculative decoding parameters from stateful decoders (#3024) 
Simplify StatefulGptDecoderBatched constructor:
  - Remove speculative decoding mode parameter
  - Initialize with default mode=None
  - Update GptSession class accordingly

Simplify setup method signatures in StatefulGptDecoder and StatefulGptDecoderBatched:
  - Remove maxTokensPerStep parameter
  - Initialize decoders with default maxTokensPerStep=1
  - Update GptSession class accordingly

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-03-26 20:16:26 +08:00
Dom Brown
f995a92a31
CI: Waive for https://nvbugspro.nvidia.com/bug/5189673 (#3100) 
* Waive for https://nvbugspro.nvidia.com/bug/5189673

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>

* Update waive

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>

---------

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-03-26 19:13:43 +08:00
Enwei Zhu
224469b096
test: [TRTLLM-4334] Create 1.0 criteria scope from API stability references (#3069) 
* committed APIs validation

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* clean name

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* separate

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* add TODOs

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix naming

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

* fix

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-03-26 18:14:35 +08:00
Kaiyu Xie
ea3739ee62
Fix: fuse message not aligned on different processes (#3067) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-03-26 17:15:27 +08:00
Zheng Duan
d70ff79d1d
conditional disagg test (#3012) 
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
 2025-03-26 15:55:33 +08:00
Ivy Zhang
3e116c9687
test: add random image test for llama-3.2-11b-vision (#3055) 
* add random image test for llama-3.2-11b-vision

Signed-off-by: Ivy Zhang <yanzh@nvidia.com>

* rename case

Signed-off-by: Ivy Zhang <yanzh@nvidia.com>

---------

Signed-off-by: Ivy Zhang <yanzh@nvidia.com>
Co-authored-by: Larry <larryx@nvidia.com>

CI got Passed: https://nv/trt-llm-cicd/job/helpers/job/PR_Github/522/
 2025-03-26 15:38:16 +08:00
Enwei Zhu
f70b439503
bitmask v3 (#3009) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-03-26 15:21:29 +08:00
Aurelien Chartier
0ec7b5701f
chore: Handle qwen2audio inputs ids expansion during processing (#3080) 
* Handle qwen2audio inputs ids expansion during processing

Signed-off-by: Aurelien Chartier <achartier@nvidia.com>

* remove more dead code

Signed-off-by: Aurelien Chartier <achartier@nvidia.com>

* fix yapf

Signed-off-by: Aurelien Chartier <achartier@nvidia.com>

---------

Signed-off-by: Aurelien Chartier <achartier@nvidia.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-03-26 15:00:27 +08:00
Yechan Kim
3c7cb6629c
Add EXAONE-Deep (#3054) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-03-26 14:24:04 +08:00
kxdc
e6cb34d921
test: fix QA TRT integration testlist mismatch issue (#3090) 
Incorrect test-db context caused empty test list output.
Fix by typo correction: `llm_trt_*` -> `trt_llm_*`.

Signed-off-by: kxdc <xink@nvidia.com>
 2025-03-26 14:03:21 +08:00
peaceh-nv
5e272eef81
feat : reduce trt engine build time in testing (#3014) 
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
 2025-03-26 13:02:54 +08:00
DylanChen-NV
1ac0566a93
fix: fix for cp > kvHeadNum (#3002) 
* fix for cp > kvHeadNum

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

* fix for None kv_head_num

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>

---------

Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
 2025-03-26 12:39:02 +08:00
HuiGao-NV
25f2434495
fix: Set correct draft_token_nums to dummy requests for torch compilation with MTP (#3053) 
Set correct draft_token_nums to dummy requests for torch compilation with MTP

Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-03-26 11:32:57 +08:00
yuxianq
268933b5cc
Refactor imports inside tensorrt_llm._torch. (#3015) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-03-26 11:01:07 +08:00
tburt-nv
e68749ca1e
2025-03-25 update CI allowlist (#3074) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
Co-authored-by: juney-nvidia <143764042+juney-nvidia@users.noreply.github.com>
 2025-03-26 08:13:01 +08:00
Anurag Mukkara
7361c7d401
Add second possible output (#3043) 
Signed-off-by: Anurag Mukkara <amukkara@nvidia.com>
 2025-03-25 12:59:27 -07:00
Enwei Zhu
f93ac9672e
clean (#3061) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-03-25 21:55:08 +08:00
Shunkangz
8ee840159b
Add updateKVCacheTransfer (#2984) 
Add kv cache transfer measurement
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-03-25 21:45:35 +08:00
Chuang Zhu
110c6fc0f0
wait long time for disagg test (#2998) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-03-25 20:52:38 +08:00
Yuan Tong
53adb3cb4e
test: waive flaky test_kv_cache_event_async_api (#3062) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-03-25 18:41:30 +08:00
Xiaowei Wang
d9acce72bb
doc: Update DeepSeekV3 doc (#3052) 
* Update DeepGEMM and flashMLA related content

* Add single-node command for deepgemm

* Fix spelling

---------

Signed-off-by: xiaoweiw-nv <100599594+xiaoweiw-nv@users.noreply.github.com>
 2025-03-25 18:17:26 +08:00
Perkz Zheng
e9df23f815
fix: [MLA] fix the bug with fp8 MLA kernels on Blackwell. (#3008) 
* update cubins
* update error message

---------

Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>
 2025-03-25 18:03:29 +08:00
bhsueh_NV
5724c61934
chore: fix bug of model paths in confset.py (#3011) 
* fix bugs of model paths of models in examples/models/contrib/

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix bug of code layout

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* fix bug of test_multimodal.py

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

* add gptj_example_root back

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>

---------

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-03-25 17:00:44 +08:00
xiweny
aacb8d66f4
doc: document running CI stage locally (#3060) 
Signed-off-by: Xiwen Yu <xiweny@nvidia.com>
 2025-03-25 16:18:17 +08:00
QI JUN
a8ec1cc4ea
remove examples/test_gptj.py::test_llm_gptj_fp8_manage_weights_summary test case (#3057) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-03-25 15:41:27 +08:00
Yan Chunwei
69feafc947
fix: amend the test list (#3056) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-03-25 14:17:36 +08:00
WeiHaocheng
7ac04ada2a
doc: Add README.md for scaffolding (#3048) 
* Add README.md for scaffolding

Signed-off-by: fredw <20514172+WeiHaocheng@users.noreply.github.com>

* Update tensorrt_llm/scaffolding/README.md

Co-authored-by: dongxuy04 <78518666+dongxuy04@users.noreply.github.com>
Signed-off-by: WeiHaocheng <20514172+WeiHaocheng@users.noreply.github.com>

---------

Signed-off-by: fredw <20514172+WeiHaocheng@users.noreply.github.com>
Signed-off-by: WeiHaocheng <20514172+WeiHaocheng@users.noreply.github.com>
Co-authored-by: dongxuy04 <78518666+dongxuy04@users.noreply.github.com>
 2025-03-25 13:58:01 +08:00
bhsueh_NV
ed84f8f923
fix bug of test_phi (#3050) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-03-25 13:12:06 +08:00
Aurelien Chartier
ef78518310
Only gather responses on rank 0 (#3040) 
Signed-off-by: Aurelien Chartier <achartier@nvidia.com>
 2025-03-24 21:54:51 -07:00
Aurelien Chartier
a33c595c88
Fix logits dtype in assert (#3038) 
Remove extra methods in trtGptModelInflightBatching.h. The methods were moved out of that class during a previous refactoring, but the definitions have been left behind.

Signed-off-by: Aurelien Chartier <achartier@nvidia.com>
 2025-03-25 10:35:21 +08:00
Zhanrui Sun
c2ffce7dbd
chore: bump version to "0.19.0.dev2025032500" (#3019) 
Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-03-25 10:04:17 +08:00
Yan Chunwei
c29cebf79d
Deprecate model_api examples (#2999) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-03-25 09:37:20 +08:00
bhsueh_NV
11f9ecb2fd
chore: remove useless param (#3023) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-03-25 08:36:45 +08:00