kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
3,297 Commits 25 Branches 59 Tags 2.1 GiB
e5865de518
Commit Graph

105 Commits

Author SHA1 Message Date
Lizhi Zhou
23d5280a90
[TRTLLM-7843][feat] implement disagg cluster auto-scaling (#8215) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-10-21 17:25:07 -04:00
Pengyun Lin
a4227cf1b0
[None][feat] Support Qwen3 reasoning parser (#8000) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-10-21 14:08:39 +08:00
Wangjue Yao
9865d3d770
[None][feat] Support cached tokens for Openai server (#7637) 
Signed-off-by: wjueyao <wyao123@terpmail.umd.edu>
Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-10-16 20:51:37 +08:00
Lizhi Zhou
22471ecc67
[TRTLLM-7846][feat] implement etcd storage for disagg cluster (#8210) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-10-14 16:48:41 -04:00
Yilin Fan
2695d70d42
[None][feat] Add request timing breakdown option in benchmark_serving (#8128) 
Signed-off-by: nv-yilinf <206948969+nv-yilinf@users.noreply.github.com>
 2025-10-10 09:24:54 -07:00
Lizhi Zhou
fdf29ab8fa
[TRTLLM-7846][feat] Http disagg-cluster management implemention (#7869) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-10-09 09:44:01 +08:00
mpikulski
98b3af4d4e
[TRTLLM-8413][chore] resolve sampling defaults in OpenAI API backend (#8121) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-10-06 06:09:43 -07:00
Yilin Fan
01423ac183
[None][feat] perf_metrics endpoint functionality improvement (#8005) 
Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>
Signed-off-by: nv-yilinf <206948969+nv-yilinf@users.noreply.github.com>
 2025-10-02 17:43:25 -07:00
Guoming Zhang
202bed4574 [None][chroe] Rename TensorRT-LLM to TensorRT LLM for source code. (#7851) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-09-25 21:02:35 +08:00
Enwei Zhu
a1a57e83b8
[TRTLLM-5235][feat] Enable regex and EBNF grammar in trtllm-serve (#7925) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-09-24 18:30:23 +08:00
JunyiXu-nv
6654b78c94
[https://nvbugs/5521799][fix] Trim incorrectly generated harmony messages (#7849) 
Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
 2025-09-24 16:38:43 +08:00
Lizhi Zhou
7550251988
[TRTLLM-7182][test] add multi-nodes test for disagg-serving (#7470) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-09-24 08:31:56 +08:00
Yilin Fan
7d4d6cc9e0
[TRTLLM-7292][feat] Support multi-threaded tokenizers for trtllm-serve (cherry-pick) (#7776) 
Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>
 2025-09-23 09:39:47 -07:00
Tailing Yuan
740340dd17
[https://nvbugs/5522847][fix] Disable GC on disagg server and client (#7858) 
Signed-off-by: Tailing Yuan <yuantailing@gmail.com>
 2025-09-23 09:16:55 +08:00
Enwei Zhu
8330d5363a
[TRTLLM-8209][feat] Support new structural tag API (upgrade XGrammar to 0.1.25) (#7893) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-09-23 09:10:09 +08:00
Yechan Kim
f77aca9f2c
[TRTLLM-7385][feat] Optimize Qwen2/2.5-VL performance (#7250) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-09-22 03:40:02 -07:00
Kaiyu Xie
6eef19297f
[None] [chore] cherry pick changes on slurm scripts from release/1.1.0rc2 (#7750) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-09-16 16:07:13 +08:00
Zheng Duan
24fc1f9acf
[None][fix] using arrival time in llmapi when creating LlmRequest in pytorch workflow (#7553) 
Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>
 2025-09-15 07:26:01 -04:00
Pengyun Lin
c2bc39af63
[TRTLLM-1302][feat] Topk logprobs for TRT backend and top1 logprob for PyT backend (#6097) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-09-12 15:32:34 +08:00
zhanghaotong
96af324ff1
[None][fix] Add try-catch in stream generator (#7467) 
Signed-off-by: Zhang Haotong <zhanghaotong.zht@antgroup.com>
Co-authored-by: Zhang Haotong <zhanghaotong.zht@antgroup.com>
 2025-09-08 16:09:26 -04:00
dominicshanshan
c9dca69e1b
[None][chore] Mass integration of release/1.0 - 3rd (#7519) 
Signed-off-by: Nave Assaf <nassaf@nvidia.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Signed-off-by: qqiao <qqiao@nvidia.com>
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
Signed-off-by: Bo Deng <deemod@nvidia.com>
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Pamela <179191831+pamelap-nvidia@users.noreply.github.com>
Signed-off-by: Hui Gao <huig@nvidia.com>
Signed-off-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com>
Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>
Signed-off-by: Michal Guzek <mguzek@nvidia.com>
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
Co-authored-by: Nave Assaf <55059536+Naveassaf@users.noreply.github.com>
Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
Co-authored-by: Emma Qiao <qqiao@nvidia.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
Co-authored-by: Bo Deng <deemod@nvidia.com>
Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
Co-authored-by: yifeizhang-c <219273404+yifeizhang-c@users.noreply.github.com>
Co-authored-by: amitz-nv <203509407+amitz-nv@users.noreply.github.com>
Co-authored-by: Erin <14718778+hchings@users.noreply.github.com>
Co-authored-by: chenfeiz0326 <chenfeiz@nvidia.com>
Co-authored-by: ChristinaZ <83400082+ChristinaZ@users.noreply.github.com>
Co-authored-by: Venky <23023424+venkywonka@users.noreply.github.com>
Co-authored-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com>
Co-authored-by: HuiGao-NV <huig@nvidia.com>
Co-authored-by: milesial <milesial@users.noreply.github.com>
Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>
Co-authored-by: Michal Guzek <moraxu@users.noreply.github.com>
Co-authored-by: peaceh-nv <103117813+peaceh-nv@users.noreply.github.com>
Co-authored-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Co-authored-by: pcastonguay <55748270+pcastonguay@users.noreply.github.com>
Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com>
Co-authored-by: Linda <57756729+Linda-Stadter@users.noreply.github.com>
Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Co-authored-by: Jiagan Cheng <jiaganc@nvidia.com>
Co-authored-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-09-08 14:03:04 +08:00
JunyiXu-nv
504bb7ffa9
[TRTLLM-7779][feat] Support multiple postprocess workers for chat completions API (#7508) 
Signed-off-by: Junyi Xu 
Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-09-08 11:11:35 +08:00
Chang Liu
23500b55c3
[TRTLLM-7398][feat] Support KV cache salting for secure KV cache reuse (#7106) 
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
 2025-09-06 17:58:32 -04:00
Rashid Kaleem
89889fb526
[https://nvbugs/5369366] [fix] Report failing requests (#7060) 
Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>
 2025-09-04 12:56:23 -07:00
JunyiXu-nv
eefe5f2093
[TRTLLM-7208][feat] Implement basic functionalities for Responses API (#7341) 
Signed-off-by: Junyi Xu <junyix@nvidia.com>
 2025-09-02 07:08:22 -04:00
Pengyun Lin
c1e7fb9042
[TRTLLM-7207][feat] Chat completions API for gpt-oss (#7261) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-08-28 10:22:06 +08:00
Iman Tabrizian
bc84758626
[None][feat] Add logging for OAI disagg server (#7232) 2025-08-26 21:02:03 -07:00
Zheng Duan
cf50ba2980
[TRTLLM-6549][feat] add perf metrics endpoint to openai server and openai disagg server (#6985) 
Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>
 2025-08-26 15:34:44 +08:00
Yechan Kim
0893afae3d
[TRTLLM-6771][feat] Support MMMU for multimodal models (#6828) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-21 08:54:12 +08:00
Chang Liu
ce53832610
[TRTLLM-7326][feat] Add standalone multimodal encoder (#6743) 
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
 2025-08-19 21:42:50 -07:00
Zero Zeng
953f4fd69e
[None][fix] acceptance rate calculation fix in benchmark_serving (#6746) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-19 17:29:36 +08:00
Bo Deng
d8acca495b
[TRTLLM-6675][infra] Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6623 (#6735) 
Signed-off-by: Bo Deng <deemod@nvidia.com>
 2025-08-14 04:36:38 +00:00
Yechan Kim
12102e2d48
[TRTLLM-6772][feat] Multimodal benchmark_serving support (#6622) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-12 19:34:02 -07:00
Zero Zeng
4b4b91ab51
[None][feat] improve dataloading for benchmark_dataset by using batch… (#6548) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-11 09:50:41 +08:00
Ye Zhang
bcf5ec0c9a
[None][feat] Core Metrics Implementation (#5785) 
Signed-off-by: Ye Zhang <zhysishu@gmail.com>
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-08-09 02:48:53 -04:00
Venky
61da2daeb4
[TRTLLM-6761][refactor] Replace LogitBiasLogitsProcessor with embedding bias tensor system (#6464) 
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
 2025-08-05 07:14:24 -07:00
Zero Zeng
48768fd720
fix: Fix missing key (#6471) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-01 14:25:58 +08:00
shaharmor98
f9cf683e39
add propagation of trust_remote_code to OpenAIServer (#6446) 
Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>
 2025-07-30 15:25:41 -04:00
Zheng Duan
c9ed1ab436
[TRTLLM-6549] chore: record delay introduced by disaggregated serving in kv cache measure (#6135) 
Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>
 2025-07-30 10:39:40 +08:00
Zero Zeng
c9b8b6180f
Add Acceptance Rate calculation to benchmark_serving (#6240) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-07-28 14:00:58 +08:00
nv-guomingz
b8d4cb8beb
feat: Support JSON Schema in OpenAI-Compatible API (#6321) 
Signed-off-by: noiji <52301388+noiji@users.noreply.github.com>
 2025-07-25 12:55:56 -04:00
xiaoqi
a0aecf0476
[feat]: support logit_bias (#5354) 
Signed-off-by: xq25478 <xq25478@qq.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: hexiao.xq <hexiao.xq@antgroup.com>
Co-authored-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Co-authored-by: hexiao.xq <hexiao.xq@antgroup.com>
Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-25 09:37:41 +00:00
Linda
9a99e6d6d7
fix: integration tests with nanobind (#6326) 
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>
 2025-07-25 09:23:20 +08:00
Rashid Kaleem
152e2df43b
[Disaggregated] Add retry knobs and handling (#5808) 
Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>
Signed-off-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>
Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>
 2025-07-19 07:27:59 +08:00
Wanli Jiang
2d2b8bae32
feat: TRTLLM-5574 Add phi-4-multimodal pytorch-backend support (#5644) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-07-17 06:30:58 +08:00
Pengyun Lin
6992616c1f [nvbug 5004744][fix] rewrite completion API to avoid repetitive tokens (#5201) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-14 17:17:30 +08:00
Kaiyu Xie
7b09a415c1
fix: Make the bench serving script compatible with different usages (#5905) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-07-10 19:36:26 +08:00
Xianjie Qiao
5ab1cf5ae6
Remove unnecessary benchmarking results (#5852) 
Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
 2025-07-09 11:19:06 +08:00
Iman Tabrizian
c508b994b6
Fix lost requests for disaggregated serving (#5815) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-07-09 08:42:45 +09:00
Yegor
b01d1c28f7
[feat] Detokenize option in /v1/completions request (#5382) 
Signed-off-by: Yegor <75512761+Wokzy@users.noreply.github.com>
Signed-off-by: Yegor Yershov <yegor6741@gmail.com>
 2025-07-08 19:36:04 +08:00