kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-22 03:35:00 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,127 Commits 28 Branches 60 Tags 2.1 GiB
f82e44bbb9
Commit Graph

27 Commits

Author SHA1 Message Date
Yilin Fan
31bb650298
Cherry pick feat/llama4 to main (#4739) 
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>
Co-authored-by: Chenfei Zhang <chenfeiz@nvidia.com>
 2025-05-30 05:28:40 +08:00
Yuxian Qiu
8f055f5d14
feat: Skip sampler for intermediate pp stages. (#4514) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-05-26 10:08:51 +08:00
Thor Johnsen
5d438be59a
[TRTLLM-5000][feat] Pytorch implementation of ngram drafter (#3936) 
* v1.5

Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>

v1.5.4 Add back draft_overhead to spec dec stats

Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>

* v1.5.5: fix CI error

Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>

* v1.6: fix CI error 8196 > 8192

Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>

* Address reviewer concerns

Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>

* Address reviewer concerns

Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>

* precommit run

Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>

* v2.0: Address reviewer concerns

Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>

* v2.1: add fix from wili

Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>

* Revert changes that require use of TypeAlias because that requires python version >= 3.10

Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>

---------

Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-05-21 10:40:00 +08:00
liji-nv
58e405624a
[https://nvbugs/5123103][fix] Fix torch compile for DeepSeekV3 (#3952) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-05-19 22:12:25 +08:00
Netanel Haber
9cd8148f28
API Breaking Change + Readability: "decoder"->"sampler" (#4121) 
* *decoder*->*sampler*; new_tensors_device: dict[str, torch.Tensor] -> device: SampleStateTensors

* **Breaking Change**, as it changes public interfaces, main changes:
* PyTorchConfig [consumed via LLM(pytorch_backend_config)]: Configuration parameters mixed_decoder and enable_trtllm_decoder -> sampler.
* Command-line argument --enable_trtllm_decoder becomes --enable_trtllm_sampler in examples/pytorch/quickstart_advanced.py.

---------

Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
 2025-05-16 23:52:25 +08:00
yuxianq
4f8afe4cc6
feat: [nvbugs/5261055][nvbugs/5170160] non-invasive pipeline parallelism (#4034) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-05-16 04:16:53 +08:00
Fanrong Li
77f8e43592
[fix] Fix relaxed acceptance to support enabling it in context phase (#4126) 
* fix relaxed acceptance to support enable this feature in context phase.

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>

* fix sample_and_accept_draft_tokens unit test.

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>

---------

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-05-09 14:11:14 +08:00
YueWeng
b1621e8d4e
feat: add relaxed acceptance for DS (#3865) 
* add relaxed acceptance for DS R1

Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>

* clean and update docs

Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>

* fix

Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>

* Modified based on review

Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>

* fix mtp manager issue

Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>

---------

Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>
Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-05-01 21:50:36 +08:00
Mike Iovine
8c2c969fcb
[fix] Pad requests to maximum draft length in spec decode (#3957) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-30 11:02:18 -04:00
Fanrong Li
e6b482ef47
fix: change the seq_lens sync copy to an async one (#3786) 
---------

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-29 23:56:49 +08:00
Perkz Zheng
35c5e4f1c5
feat: add CGA reduction fmha kernels on Blackwell. (#3763) 
* update cubins

Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>

* add trtllm-gen kernels for eagle3 and also kernels with cga-reduction

Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>

* address the comments

Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>

---------

Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>
 2025-04-29 10:43:54 +08:00
Mike Iovine
e534bf09cc
[fix] Fix flashinfer + speculation issues (#3686) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-28 14:34:22 -04:00
Yuan Tong
57944206ba
feat: return logits in PyTorch flow (#3221) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-24 16:56:03 -07:00
Fanrong Li
bc1c4ddcb5
fix: remove the unnecessary metadata changes in mtp. (#3787) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-23 16:01:28 +08:00
Mike Iovine
41a6c98544
Support CUDA graphs for EAGLE3 (#3176) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-04-17 04:53:50 +08:00
Yuan Tong
d4c0423cdb
refactor: collect executor and decoder states into dataclass (#3234) 
* fix: Proper error bubbling for PyExecutor

Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-15 16:31:45 +08:00
Fanrong Li
e8b97341de
fix the py_decoding_iter update in decoder. (#3297) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-07 11:18:33 +08:00
tburt-nv
7a659885e3
chore: remove usernames from comments (#3291) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
 2025-04-05 13:44:28 +08:00
Jinyang Yuan
1128dc2a5a
perf: Use pinned H2D to reduce bubbles (#3147) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-04-04 22:23:10 +08:00
Fanrong Li
1fe64b90be
fix: fix the acceptance rate of pytorch workflow in trtllm-bench (#3240) 
* fix acceptance rate of pytorch workflow.
* revert the RequestOutput API change.

---------

Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-03 15:12:24 +08:00
dongjiyingdjy
22ff81b047
fix：fix illeagel memory access when mtp >= 2 (#3006) 
* fix - fix illeagel memory access when mtp > 2

---------

Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-01 13:36:45 +08:00
Mike Iovine
5416966ddb
Add initial EAGLE-3 implementation (#3035) 
Signed-off-by: Mike Iovine <miovine@nvidia.com>
 2025-03-29 22:31:24 +08:00
yuxianq
268933b5cc
Refactor imports inside tensorrt_llm._torch. (#3015) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-03-26 11:01:07 +08:00
Kaiyu Xie
2631f21089
Update (#2978) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-03-23 16:39:35 +08:00
Kaiyu Xie
3aa6b11d13
Update TensorRT-LLM (#2936) 
* Update TensorRT-LLM

---------

Co-authored-by: changcui <cuichang147@gmail.com>
 2025-03-18 21:25:19 +08:00
Kaiyu Xie
9b931c0f63
Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
Kaiyu Xie
77d7fe1eb2
Update TensorRT-LLM (#2849) 
* Update TensorRT-LLM

---------

Co-authored-by: aotman <chenhangatm@gmail.com>
 2025-03-04 18:44:00 +08:00