kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,529 Commits 25 Branches 59 Tags 2.1 GiB
ac0b3f8b66
Commit Graph

1529 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
xavier-nvidia
ac0b3f8b66
GEMM+AR TP2 harness fix (#6027) 2025-07-14 15:52:53 -07:00
xavier-nvidia
60538df1b5
Fix GEMM+AR nvbugs 5219533,5127801,5072306 (#5969) 
Signed-off-by: xsimmons <xsimmons@nvidia.com>
 2025-07-11 10:22:02 -07:00
Zhihan Jiang
1cf89e569b [nvbugs/5321981] Cherrypick fix: Fix the Llama3.1 405B hanging issue. (#5698) (#5925) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
Co-authored-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-07-10 22:53:10 -07:00
Iman Tabrizian
435eeea8b6 Fix lost requests for disaggregated serving (#5815) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-07-10 22:51:04 -07:00
Kaiyu Xie
155b19e6b0 feat: Return context response immediately when stream_interval > 1 (#5836) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-07-10 22:50:57 -07:00
Yegor
5b7007a69d [feat] Detokenize option in /v1/completions request (#5382) 
Signed-off-by: Yegor <75512761+Wokzy@users.noreply.github.com>
Signed-off-by: Yegor Yershov <yegor6741@gmail.com>
 2025-07-10 22:50:36 -07:00
jmydurant
8836990bde
[TRTLLM-3602][feat] support nvfp4 model and fp8 kv cache for MLA chunked prefill (Blackwell) (#5475) 
Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>
 2025-06-26 22:18:08 +08:00
Robin Kobus
8dfa31c71d
refactor: remove batch_manager::KvCacheConfig and use executor::KvCacheConfig instead (#5384) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-06-26 19:45:52 +08:00
Omer Ullman Argov
6bae76d7ca
[fix][ci] move torch tests to run under torch stage (#5473) 
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
 2025-06-26 14:31:38 +03:00
Omer Ullman Argov
1633bd2bef
[CI] move flashinfer llama tests to post merge (#5506) 
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
 2025-06-26 19:27:32 +08:00
Frank
baf7eaa1cc
Add trtllm-bench reviewers. (#5452) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-06-26 18:48:00 +08:00
Rashid Kaleem
3a1f4d4001
[feat] Add progress bar to benchmark (#5173) 
Signed-off-by: Rashid Kaleem <rkaleem@nvidia.com>
Co-authored-by: Zhihan Jiang <68881590+nvzhihanj@users.noreply.github.com>
Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com>
 2025-06-26 18:39:45 +08:00
Kaiyu Xie
2eb6502b1d
feat: Add support for TRTLLM CustomDataset (#5511) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-06-26 18:27:37 +08:00
Yao Yao
0788c5d0d6
[perf] improve XQA-MLA perf (#5468) 
Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
 2025-06-26 18:09:13 +08:00
Kaiyu Xie
749393ec9f
doc: Fix benchmark cmd in disagg scripts (#5515) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-06-26 17:04:55 +08:00
xinhe-nv
ff2dd72df4
tests: waive tests (#5458) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-06-26 14:53:55 +08:00
Omer Ullman Argov
fa0ea92dfd
[fix][ci] trigger multigpu tests for deepseek changes (#5423) 
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
 2025-06-26 14:30:00 +08:00
Bo Li
1bab9000a6
perf: Optimize swizzle_sf, unswizzle_sf, reswizzle_sf (#5318) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-06-26 14:03:56 +08:00
Alessio Netti
7e681fbe52
[chore] Allow configuring linking of NVRTC wrapper (#5189) 
Signed-off-by: Alessio Netti <netti.alessio@gmail.com>
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Co-authored-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-06-26 07:26:10 +02:00
dongxuy04
490d2e5819
feat: large-scale EP(part 8: Online EP load balancer integration for PCIe fp8) (#5226) 
Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
 2025-06-25 22:25:13 -07:00
amitz-nv
e0bb123ae7
[TRTLLM-5921][feat] Prevent serialization of entire LoRA adapters in each request (#5080) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-06-26 08:15:06 +03:00
Yukun He
9ee33605bb
[TRTLLM-6019] feat: Remove cutlass min latency code from AutoTuner. (#5394) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-06-26 13:12:03 +08:00
Daniel Stokes
942841417e
opensource: Opensource MOE MXFP8-MXFP4 implementation (#5222) 
Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>
 2025-06-26 12:18:19 +08:00
qsang-nv
e9cd810071
keep sm90 headsize 128 cubins (#5320) 
Signed-off-by: Qidi Sang <200703406+qsang-nv@users.noreply.github.com>
 2025-06-26 12:14:01 +08:00
Netanel Haber
6aef14943c
Revert "feature: unify new_tokens format sample state to trtllm samper new_tokens format (#4401)" (#5474) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
 2025-06-25 20:56:04 -07:00
Emma Qiao
32d1573c43
[Infra] - Add timeout setting for long tests found in post-merge (#5501) 
Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-06-26 11:31:39 +08:00
Venky
d9b75f83fd
[CI] Waive test_fp8_block_scales_4gpus[ep4-mtp_nextn=0-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False] (#5494) 
Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>
 2025-06-25 20:17:12 -07:00
ChristinaZ
d135f5993d
Add unit test for routing kernels (#5405) 
Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>
 2025-06-26 09:49:11 +08:00
jmydurant
578dbc8d9a
feat: chunked prefill for MLA (Blackwell) (#4651) 
Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>
 2025-06-26 09:01:00 +08:00
Yukun He
3fc57543e2
[5356427] fix: Remove the seq_len of 4096 from FP8 block scale MoE tuning configs. (#5485) 
The seq_len of 4096 will cause some unknown CUDA illegal memory access issue if run with some other tests consecutively.
Put a saturated upper bound for any sequence length larger than it.
 2025-06-26 08:38:35 +08:00
HuiGao-NV
74ae15a26b
CI: enable test cases on single device type (#5484) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-06-26 08:03:44 +08:00
Xianjie Qiao
1e4fa13d33
Add sleep function for disagg gen-only benchmarking (#5398) 
Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
 2025-06-26 07:32:16 +08:00
QI JUN
feaf789342
CI: reduce BF16 test cases in B200 (#5482) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-06-26 07:18:20 +08:00
Omer Ullman Argov
bdc8dfebc3
[fix][ci] dont build wheel for cpp tests (#5443) 
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
 2025-06-26 00:13:47 +03:00
Omer Ullman Argov
61bb71fd1b
[fix][test] remove test in global scope (#5470) 
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
 2025-06-25 23:42:26 +03:00
QI JUN
3a2c4ca77b
chore: split _build_model method for TorchLlm and TrtLlm (#5418) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-06-26 04:32:46 +08:00
Mike Iovine
5bc8c894f7
[chore] Disable block reuse when draft model speculation is being used (#5448) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-06-26 03:51:20 +08:00
Daniel Cámpora
205c97a4ae
[TRTLLM-5974][feat] Support disaggregated serving in TRTLLM Sampler (#5328) 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
Signed-off-by: Daniel Cámpora <961215+dcampora@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-06-25 17:41:36 +02:00
Kaiyu Xie
c5ae3272b9
feat: Make benchmark_serving part of the library (#5428) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-06-25 23:13:56 +08:00
HuiGao-NV
314f15f0a7
Fix: fix nvbug 5356427 (#5464) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-06-25 22:24:26 +08:00
HuiGao-NV
cc3c2b3be2
Move 3 disaggregated cases from 4 GPUs devices to 1 GPU device (#5457) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-06-25 21:38:14 +08:00
Kaiyu Xie
d6ada5ffce
[nvbug/5354956] fix: unexpected keyword argument 'streaming' (#5436) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-06-25 20:37:24 +08:00
HuiGao-NV
b3a4c1f404
feat: Remove not used padding_idx in models (#5385) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-06-25 17:19:59 +08:00
QI JUN
2901c5a5bc
CI: waive test_ad_build_small_multi (#5471) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-25 16:44:42 +08:00
Perkz Zheng
1f292ff2a0
[https://jirasw.nvidia.com/browse/TRTLLM-4645] support mutliCtasKvMode for high-throughput MLA kernels (#5426) 
Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>
 2025-06-25 16:31:10 +08:00
Yiqing Yan
f3cfe86dd1
chore: bump version to 1.0.0rc1 (#5460) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-06-25 16:21:34 +08:00
Netanel Haber
3ca2f6ac51
start OAIServer with max_beam_width=1 for TorchSampler (#5427) 
Signed-off-by: Netanel Haber <nhaber@nvidia.com>
 2025-06-25 15:52:06 +08:00
QI JUN
478f668dcc
CI: update multi gpu test triggering file list (#5466) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-06-25 15:51:02 +08:00
Enwei Zhu
fc7a81ceb0
test: Add LLGuidance test and refine guided decoding (#5348) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-06-25 14:12:56 +08:00
Enwei Zhu
76da7fed86
fix (NvBug 5354925): Fix static EPLB (#5411) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-06-25 13:14:40 +08:00