kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
1,133 Commits 25 Branches 59 Tags 2.1 GiB
a02df6aa4b
Commit Graph

1133 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yuxian Qiu
a02df6aa4b
fix: re-enable tp/pp for quickstart_advanced.py. (#4766) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-05-31 19:13:46 +08:00
Yan Chunwei
93c0632ee4
opt: the perormance for dist-agg streaming generation (#4214) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-05-31 17:40:32 +08:00
Emma Qiao
c945e92fdb
[Infra]Remove some old keyword (#4552) 
Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-05-31 13:50:45 +08:00
Mike Iovine
8cb6163a57
[fix] Fix Llama 3.3 70b EAGLE (#4772) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-05-30 10:08:08 -04:00
juney-nvidia
49f2f1f8eb
Expose new tech blog about DSR1 throughput optimization to the main R… (#4803) 
Signed-off-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
 2025-05-30 20:44:12 +08:00
Tao Li @ NVIDIA
3b7120d60e
DeepSeek R1 throughut optimization tech blog for Blackwell GPUs (#4791) 
Signed-off-by: Tao Li
 2025-05-30 18:54:19 +08:00
Yuxian Qiu
f82e44bbb9
fix: [nvbugs/5310520] disable embed_tokens's TP when DP enabled for llama model. (#4758) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-05-30 18:04:08 +08:00
Pengyun Lin
bac22ff7b5
[feat] support sharegpt downloading in benchmark_serving (#4578) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-05-30 17:27:53 +08:00
QI JUN
99fdef20c4
[TRTLLM-5516] perf: replicate dummy request for cuda graph padding (#4729) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-05-30 17:14:23 +08:00
ixlmar
c026dda400
fix: iteration logging and typing in PyExecutor (#4734) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-05-30 11:01:20 +02:00
ixlmar
7e6d06d5d7
feat: estimate GPU mem. usage w/ minimal KV cache (#4574) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-05-30 10:40:45 +02:00
Zheng Duan
54200ee8ac
fix: random fail of cache router test (#4597) 
Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com>
 2025-05-30 16:28:19 +08:00
Chuang Zhu
f117d6abe9
Fabric Memory for KV Cache Transfer (#4717) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-05-30 15:50:21 +08:00
Enwei Zhu
ee916da8f1
test: Waive test_llm_loading_from_ckpt_for_tp2 (#4797) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-05-30 15:43:00 +08:00
xinhe-nv
53794b26f8
test: skip test_llm_hf_gemma_quantization_1gpu_vswa on A100 (#4779) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-05-30 15:12:12 +08:00
Thor Johnsen
55d56f8155
[JIRA-5226219][fix] Fix Bug in KV cache manager (#4596) 
Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>
 2025-05-29 22:03:20 -07:00
Aurelien Chartier
36b87b8671
chore: fix llm_root when LLM_ROOT is not set (#4741) 
Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
 2025-05-29 19:44:34 -07:00
juney-nvidia
fe359d9df9
Added code owners for AutoDeploy (#4769) 
Signed-off-by: Jun Yang <143764042+juney-nvidia@users.noreply.github.com>
 2025-05-30 09:55:27 +08:00
Jinyang Yuan
5339d367ce
[perf] Reduce the workspace size of FP4 activation scales for MoE (#4303) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-05-30 09:03:52 +08:00
hlu1
3093c747b7
[Architecture] Redesign Linear module (#4721) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com@users.noreply.github.com>
Co-authored-by: Hao Lu <14827759+hlu1@users.noreply.github.com@users.noreply.github.com>
 2025-05-29 16:05:46 -07:00
Yilin Fan
31bb650298
Cherry pick feat/llama4 to main (#4739) 
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>
Co-authored-by: Chenfei Zhang <chenfeiz@nvidia.com>
 2025-05-30 05:28:40 +08:00
Robin Kobus
79a94a28f9
refactor: unique_ptr instead of shared_ptr (#4697) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-05-29 22:49:35 +02:00
Mike Iovine
2d61174a9b
[feat] Support RULER + chunked prefill in lm-eval-harness (#4592) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-05-30 04:14:00 +08:00
Jhao-Ting Chen
fcadce9f8d
[fix] Eagle-2 LLMAPI pybind argument fix. (#3967) 
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
 2025-05-29 12:23:25 -07:00
QI JUN
255779a91d
Chore: fuse _merge_requests method into _fetch_new_requests method (#4689) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-05-29 17:47:44 +08:00
yuanjingx87
2c48ff5898
[feat] add b200 support via slurm (#4709) 
Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com>
 2025-05-29 14:49:46 +08:00
Yan Chunwei
33a9ba55f5
fix: test trtllm-bench mgmn (#4613) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-05-29 14:43:47 +08:00
ruodil
500aca4f44
test: remove perf test l40s/l20 oom test cases and unwaive tests (#4755) 
Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com>
 2025-05-29 13:58:47 +08:00
Zhanrui Sun
7b2b657198
infra: [TRTLLM-5247][TRTLLM-5248][TRTLLM-5249] Refactor docker build image groovy and support NGC images (#4294) 
Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
Signed-off-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-05-29 11:23:29 +08:00
QI JUN
058f83e47b
CI: move post-merge multi GPU test of PyTorch backend to H200 (#4733) 
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-05-29 11:15:56 +08:00
Yiqing Yan
7f29a70f53
Waive L0 test (#4748) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-05-29 11:05:27 +08:00
Yan Chunwei
ac17142495
chore: rename ExecutorBindingsWorker/Proxy (#4716) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-05-29 10:32:35 +08:00
yuanjingx87
2307e91122
[fix] add back rtx6000pro tests (#4679) 
Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com>
 2025-05-29 09:22:52 +08:00
Arthur Rasmusson
812b1abf86
feature: KV Cache GPUDirect Storage (#3209) 
Signed-off-by: Arthur Rasmusson <47877520+arthurrasmusson@users.noreply.github.com.>
Co-authored-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Co-authored-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
 2025-05-28 23:27:43 +00:00
Erin
820c39041f
chore: [nvbug_5273941] unwaive test_llm_loading_from_ckpt_for_tp2 (#4725) 
Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
 2025-05-29 06:54:32 +08:00
Yuxian Qiu
bf691b3d28
feat: support packed weights in vanilla moe (#4719) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-05-29 06:24:24 +08:00
Aurelien Chartier
6cf1e4d0a9
chore: add -f to pkill calls (#4711) 
Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
 2025-05-29 02:54:31 +08:00
Robin Kobus
12763779c4
chore: Clean up cpp runtime (#4449) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-05-28 16:32:59 +02:00
Ivy Zhang
ed3c67e34a
tests: [https://nvbugspro.nvidia.com/bug/5289908] run maverick bf16 on blackwell (#4722) 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
 2025-05-28 22:05:51 +08:00
xinhe-nv
93283484c2
test: [CI] Add failed cases into waives.txt (#4688) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-05-28 22:04:35 +08:00
Aurelien Chartier
6b96f09ff2
chore: remove extra paths to find binaries (#4706) 
Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
 2025-05-28 06:51:42 -07:00
Yan Chunwei
5506f60037
chore [BREAKING CHANGE]: Flatten PyTorchConfig knobs into TorchLlmArgs (#4603) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-05-28 18:43:04 +08:00
ixlmar
fbe4db207d
feat: forward exceptions to Python and catch OOMs (#4497) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-05-28 11:58:10 +02:00
Yiqing Yan
66828015a0
Fix rerun step (#4715) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-05-28 17:01:05 +08:00
Iman Tabrizian
c875184f78
Add missing serialization classes (#4642) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-05-28 16:40:23 +08:00
amirkl94
fbec0c3552
Release 0.20 to main (#4577) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Signed-off-by: Martin Marciniszyn Mehringer <11665257+MartinMarciniszyn@users.noreply.github.com>
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Ruodi <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
Signed-off-by: Simeng Liu <simengl@nvidia.com>
Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
Signed-off-by: moraxu <mguzek@nvidia.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Co-authored-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Co-authored-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
Co-authored-by: Martin Marciniszyn Mehringer <11665257+MartinMarciniszyn@users.noreply.github.com>
Co-authored-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
Co-authored-by: Yukun He <23156053+hyukn@users.noreply.github.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Co-authored-by: Venky <23023424+venkywonka@users.noreply.github.com>
Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com>
Co-authored-by: stnie <82932102+stnie@users.noreply.github.com>
Co-authored-by: Simeng Liu <109828133+SimengLiu-nv@users.noreply.github.com>
Co-authored-by: Faraz <58580514+farazkh80@users.noreply.github.com>
Co-authored-by: Michal Guzek <moraxu@users.noreply.github.com>
Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
Co-authored-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-05-28 16:25:33 +08:00
Kaiyu Xie
b800adc65c
Fix: hang on disagg when MNNVL two-shot AllReduce is enabled (#4678) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-05-28 13:03:53 +08:00
Martin Marciniszyn Mehringer
f3fba4cc63
doc: Document the docker release image on NGC (#4705) 
Signed-off-by: Martin Marciniszyn Mehringer <11665257+MartinMarciniszyn@users.noreply.github.com>
 2025-05-28 12:00:53 +08:00
Pengyun Lin
971d16a2ee
[TRTLLM-1658][feat] Enable multiple response in trtllm-serve for TRT backend (#4623) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-05-28 11:36:44 +08:00
Bo Li
9c4b8f66b4
feat: Integration of Fused QKNorm+RoPE. (#4611) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-05-28 11:20:45 +08:00