kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-02-14 06:53:50 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
3,132 Commits 28 Branches 68 Tags 2.1 GiB
72fcff1044
Commit Graph

1375 Commits

Author SHA1 Message Date
amitz-nv
a1e03af0f4
[TRTLLM-7346][fix] Improve performance of PyTorchModelEngine._get_lora_params_from_requests (#7033) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-08-25 10:37:40 +03:00
Enwei Zhu
be6d92f09f
[None][fix] Fix MoE load balancer config loading (#7150) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-08-25 01:42:54 -04:00
Yukun He
9c5b464fe0
[None][feat] Apply AutoTuner to fp8_block_scale_deep_gemm to trigger JIT ahead of time. (#7113) 
Because deep_gemm.gp8_gemm_nt will trigger many JIT processes during the inference phase, we need to sweep these shapes ahead of time. Apply the AutoTuner framework to achieve this and retain the potential capability to tune the swap_ab flag.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-08-25 10:48:31 +08:00
ajrasane
068056677f
[None][chore] Enable auto deploy accuracy test in CI (#7179) 
Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
 2025-08-24 08:42:30 -07:00
dongxuy04
19a0ea363b
[TRTLLM-6743][feat] Optimize and refactor alltoall in WideEP (#6973) 
Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
Signed-off-by: Dongxu Yang <dongxuy@nvidia.com>
Co-authored-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
 2025-08-24 08:15:29 -04:00
amitz-nv
35e0ae484a
[https://nvbugs/5467232][fix] Fix load_torch_hf_lora to override lora_config.trtllm_modules_to_hf_modules with default only when it has no value (#7132) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-08-24 15:00:24 +03:00
Grace Ho
3d54a1a521
[None] [feat] nsys profile output kernel classifier (#7020) 
Signed-off-by: Grace Ho <grho@nvidia.com>
 2025-08-23 00:57:37 -04:00
Frank
81fd468fec
[None][fix] Correct KV cache percentage report out. (#7102) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-08-22 10:28:57 -07:00
Izzy Putterman
b36460d7b5
[None][feat] Deepseek: Start Eagle work (#6210) 
Signed-off-by: Izzy Putterman <iputterman@nvidia.com>
Co-authored-by: Mike Iovine <miovine@nvidia.com>
 2025-08-22 12:57:17 -04:00
tomeras91
c232ba8157
[TRTLLM-4921][feat] Enable chunked prefill for Nemotron-H (#6334) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
Signed-off-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-08-22 12:15:20 -04:00
Suyog Gupta
e3de5758a3
[#7136][feat] trtllm-serve + autodeploy integration (#7141) 
Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
 2025-08-22 08:30:53 -07:00
Yiqing Yan
907bc22fcb
[None][chore] Bump version to 1.1.0rc2 (#7167) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-08-22 22:02:28 +08:00
Daniel Cámpora
099f081e03
[TRTLLM-7155][feat] Unify sampler handle logits implementation. (#6867) 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
 2025-08-22 08:09:30 +02:00
Yukun He
983dd7e57c
[None][fix] Fix mm_placholder_counts extraction issue. (#7118) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-08-22 12:28:30 +08:00
Wanli Jiang
07c711eb1f
[TRTLLM-6825][fix] Update lora for phi4-mm (#6817) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-08-21 22:00:04 -04:00
dominicshanshan
6f245ec78b
[None][chore] Mass integration of release/1.0 (#6864) 
Signed-off-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com>
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com>
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Bo Deng <deemod@nvidia.com>
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
Signed-off-by: qqiao <qqiao@nvidia.com>
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
Signed-off-by: raayandhar <rdhar@nvidia.com>
Co-authored-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com>
Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com>
Co-authored-by: Yiqing Yan <yiqingy@nvidia.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>
Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com>
Co-authored-by: Bo Deng <deemod@nvidia.com>
Co-authored-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
Co-authored-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
Co-authored-by: Emma Qiao <qqiao@nvidia.com>
Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Co-authored-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com>
Co-authored-by: Raayan Dhar <58057652+raayandhar@users.noreply.github.com>
Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>
 2025-08-22 09:25:15 +08:00
Daniel Stokes
f7c597ec40
[None][perf] Make finalize fusion part of the tactic selection logic (#6915) 
Signed-off-by: djns99 <40156487+djns99@users.noreply.github.com>
 2025-08-21 14:08:03 -07:00
Fridah-nv
e18dacc931
[#4403][refactor] Move fusion, kvcache, and compile to modular inference optimizer (#7057) 
Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
 2025-08-21 10:30:36 -07:00
ChristinaZ
c7269ea93a
[https://nvbugs/5392414] [fix] Add customized default routing method (#6818) 
Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>
 2025-08-21 16:58:41 +08:00
Fridah-nv
647a52698a
[https://nvbugs/5443039][fix] Fix AutoDeploy pattern matcher for torch 2.8 (#7076) 
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
 2025-08-21 01:14:51 -04:00
Chang Liu
75b8a90816
[None][fix] Fix llama4 multimodal by skipping request validation (#6957) 
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
 2025-08-20 21:58:53 -04:00
Yechan Kim
0893afae3d
[TRTLLM-6771][feat] Support MMMU for multimodal models (#6828) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-21 08:54:12 +08:00
Robin Kobus
b95cab2a7c
[None][ci] move unittests to sub-directories (#6635) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-08-20 05:42:22 -04:00
Chang Liu
ce53832610
[TRTLLM-7326][feat] Add standalone multimodal encoder (#6743) 
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
 2025-08-19 21:42:50 -07:00
Fridah-nv
c02592d051
[None][autodeploy] Add group attention pattern for solar-pro-preview (#7054) 
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
 2025-08-19 18:57:09 -04:00
Jinyang Yuan
0e30fe4372
[None][fix] Fix assertion errors of quantization when using online EPLB (#6922) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-08-19 11:28:36 -07:00
Michal Guzek
7334f9390c
[None][fix] Accommodate Phi3/4 to work with ModelOpt's FP8 ckpts in Torch (#6761) 
Signed-off-by: Michal Guzek <mguzek@nvidia.com>
 2025-08-19 09:22:46 -07:00
zhhuang-nv
7e135d2ea7
[None][feat] Use Separate QKV Input Layout for Context MLA (#6538) 
Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>
 2025-08-19 22:04:48 +08:00
Zero Zeng
953f4fd69e
[None][fix] acceptance rate calculation fix in benchmark_serving (#6746) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-19 17:29:36 +08:00
Shunkangz
54ec2c1af1
[None][opt] Add batch wait timeout in fetching requests (#6923) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-08-19 03:50:08 -04:00
Yi Zhang
a15af879ec
[None][refactor] Refactor Torch Compile Backend, MoeLoadBalancer and warmup Logic (#6615) 
Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
 2025-08-19 09:58:44 +08:00
Daniel Cámpora
d16af87d03
[TRTLLM-7158][feat] Introduce sampler options in trtllm bench (#6855) 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
 2025-08-18 18:10:05 -04:00
Kaiyu Xie
e88cb92f24
[None] [feat] Support accurate device iter time (#6906) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-08-18 13:47:14 +08:00
bhsueh_NV
85cbd0263b
[None][feat] Support Yarn on Qwen3 (#6785) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-08-17 07:21:29 +08:00
Izzy Putterman
f6ff0e3311
[None][fix] Skip Topk if 0 (#6934) 
Signed-off-by: Izzy Putterman <iputterman@nvidia.com>
 2025-08-16 02:17:36 -04:00
Daniel Cámpora
53312eeebd
[TRTLLM-7157][feat] BREAKING CHANGE Introduce sampler_type, detect sampler according to options (#6831) 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
 2025-08-16 00:27:24 -04:00
Yiqing Yan
ec3d9f8052
[None][chore] Bump version to 1.1.0rc1 (#6953) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-08-16 10:32:47 +08:00
Yuening Li
1f8ae2b2db
[TRTLLM-5863][feat] Support MoE INT8 Weight-Only-Quantization in PyTorch Workflow (#6629) 
Signed-off-by: Yuening Li <62227368+yueningl@users.noreply.github.com>
 2025-08-15 17:15:49 -04:00
dongfengy
0ad0b967bb
[None][fix] Make TP working for Triton MOE (in additional to EP we are using) (#6722) 
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
 2025-08-15 16:58:42 -04:00
ajrasane
4162d2d746
[None][test] Add accuracy evaluation for AutoDeploy (#6764) 
Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
 2025-08-15 13:46:09 -04:00
yifeizhang-c
4127d77678
[https://nvbugs/5394392][fix] Enlarge scheduler capacity under disagg bs == 1 (#6537) 
Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>
 2025-08-15 09:52:06 -07:00
liji-nv
18ccd053d3
[https://nvbugs/5427801][fix] Torch compile support for Llama4 and Ea… (#6858) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-08-15 11:14:20 -04:00
tomeras91
f7dbc1435a
[None] [chore] Mamba cache in separate file (#6796) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
 2025-08-15 13:42:51 +03:00
Bo Li
15aabc1540
[None][fix] Fix perfect router. (#6797) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-08-14 20:09:08 -07:00
Frank
2cc59aacb3
[None][fix] Correct reporting of torch_dtype for ModelConfig class. (#6800) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-08-14 22:46:20 -04:00
qianbiao
5c2f0fd03d
[None] [feat] Add Tencent HunYuanMoEV1 model support (#5521) 
Signed-off-by: sorenwu <sorenwu@tencent.com>
Co-authored-by: sorenwu <sorenwu@tencent.com>
Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>
 2025-08-15 06:56:44 +08:00
Mike Iovine
078e907b16
[https://nvbugs/5455651][fix] Make ngram use XQA attention on Blackwell (#6873) 
Signed-off-by: Michael Iovine <miovine@nvidia.com>
Signed-off-by: Mike Iovine <miovine@nvidia.com>
Signed-off-by: Mike Iovine <mike.iovine7@gmail.com>
 2025-08-14 18:36:19 -04:00
Bo Li
26f413ad90
[https://nvbugs/5450262][fix] Fix unsupported alltoall use case (#6882) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-08-14 17:46:54 -04:00
Matthias Jouanneaux
69574ad730
[TRTLLM-5966][feat] Helix: extend mapping to support different CP types (#6816) 
Signed-off-by: Matthias Jouanneaux <mjoux@nvidia.com>
 2025-08-14 09:00:02 -07:00
kris1025
4aed7a7d19
[TRTLLM-6853][feat] refactor deepseekv3 model (#6698) 
Signed-off-by: linquanh <linquanh@nvidia.com>
 2025-08-14 11:03:17 -04:00
Pengbo Wang @ NVIDIA
ffc976ceaf
[https://nvbugs/5445466][fix] fix deepseek r1 hang by not enabling mnnvl by default (#6860) 
Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>
Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com>
 2025-08-14 22:36:56 +08:00
Shi Xiaowei
1095dfd03c
[None][fix] BREAKING CHANGE: Mismatch between docs and actual commands (#6323) 2025-08-14 03:48:57 -04:00
Yan Chunwei
0132c1db84
[https://nvbugs/5427043][fix] request length exceeds max_num_tokens (#6821) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-08-14 13:31:12 +08:00
Bo Deng
d8acca495b
[TRTLLM-6675][infra] Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6623 (#6735) 
Signed-off-by: Bo Deng <deemod@nvidia.com>
 2025-08-14 04:36:38 +00:00
jmydurant
4200fa46d1
[None][feat] Add support for Hopper MLA chunked prefill (#6655) 
Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>
 2025-08-14 10:39:26 +08:00
Izzy Putterman
ef53de8eef
[None][feat] Add test for speculative rejection sampler (2-model) (#6542) 
Signed-off-by: Izzy Putterman <iputterman@nvidia.com>
 2025-08-13 22:09:35 -04:00
Tin-Yin Lai
6c52bb07ff
[https://nvbugs/5302040][feat] Add whisper support (Bert Attention on SM100 and GPTAttention for cross attention on SM100) (#5527) 
Signed-off-by: tinyinl <tinyinl@nvidia.com>
 2025-08-13 11:19:13 -07:00
danielafrimi
bda42f8c3a
[None][feat] Support running heterogeneous model execution for Nemotron-H (#6866) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
 2025-08-13 19:51:19 +03:00
Anthony Chang
2198587b35
[https://nvbugs/5378031] [feat] Hopper W4A8 MoE supports ModelOpt ckpt for PyT backend (#6200) 
Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>
 2025-08-13 21:24:40 +08:00
Yukun He
bc5f766e0e
[TRTLLM-4501][feat] AutoTuner tuning config refactor and valid tactic generalization. (#6545) 
* Generalize the definition of tactics so that users can implement more customizable tactic types, making the configurations clearer for each kernel run. 
* Allow the user not to specify the `gen_tuning_buckets` or the `map_to_tuning_buckets` function.
* Other code refactoring.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-08-13 16:25:22 +08:00
Void
1d80df0955
[None][feat] DeepEP LL combine FP4 (#6822) 
Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>
 2025-08-13 04:20:21 -04:00
Mike Iovine
f68e03e646
[https://nvbugs/5452167][fix] Fix ngram padding issue (#6837) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-08-13 11:23:16 +08:00
Yechan Kim
12102e2d48
[TRTLLM-6772][feat] Multimodal benchmark_serving support (#6622) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-12 19:34:02 -07:00
Fanrong Li
1bbc0e323b
[None][fix] Pre-allocate workspaces for DeepGEMM MoE to avoid frequent cudaFree/cudaMalloc (#6811) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-08-13 10:27:57 +08:00
rakib-hasan
2923eb88a1
[None][fix] Refactoring input prep to allow out-of-tree models (#6497) 
Signed-off-by: Rakib Hasan <rhasan@nvidia.com>
 2025-08-12 20:29:10 -04:00
dongxuy04
bd9a6dd9ab
[TRTLLM-7008][fix] fix wideEP weights loading and args (#6789) 
Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
 2025-08-12 19:14:20 -04:00
Robin Kobus
45c7518032
[None][refactor] Simplify decoder state initialization (#6559) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-08-12 21:44:41 +02:00
Robin Kobus
dd11e08d26
[#6187][feat] add LayerNorm module (#6625) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-08-12 21:43:30 +02:00
nvchenghaoz
81f0ded1c4
[None][feat] Add GPT OSS support for AutoDeploy (#6641) 
Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>
 2025-08-12 14:03:22 -04:00
Jhao-Ting Chen
a060e12041
[https://nvbugs/5438869][fix] Set nvfp4 expert w1 w3 weight scale to the same value if they're not (#6656) 
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
 2025-08-12 20:47:10 +08:00
Shunkangz
ab0d768acf
[None][fix] Fix attention dp log (#6570) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-08-12 04:53:09 -04:00
Liao Lanyu
f7c13a4aa7
[TRTLLM-6906][chore] Using pybind to bind functions in thop/attentionOp (#6745) 
Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>
 2025-08-12 16:45:16 +08:00
Sergey Klevtsov
27fc35175e
[None][feat] CUTLASS MoE FC2+Finalize fusion (#3294) 
Signed-off-by: Sergey Klevtsov <sklevtsov@nvidia.com>
 2025-08-12 15:56:48 +08:00
Fridah-nv
0dc4b4e699
[#4403][autodeploy] Refactor: Move more transformations to new inf optimizer, Add quantization_source to factory interface (#6760) 
Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
 2025-08-11 22:02:46 -07:00
Enwei Zhu
7c686ba8de
[TRTLLM-2285][feat] Enable guided decoding with CUDA graph padding and draft model chunked prefill (#6774) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-08-12 09:30:06 +08:00
Ziyi Xiong
b4fcd5f592
[https://nvbugs/5441438][fix] Set correct draft length for the cuda graph dummy request (#6701) 
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
 2025-08-12 09:28:47 +08:00
Jinyang Yuan
ead89a0e40
[None][perf] Improve the performance of online EPLB on Hopper by better overlapping (#6624) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-08-12 09:25:13 +08:00
Chang Liu
be9dd4713c
[https://nvbugs/5385987][fix] Fix Qwen2 quantization issue by pinning transformers version (#6673) 
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>
 2025-08-11 17:16:49 -07:00
rakib-hasan
7ab8112450
[None][fix] Refactoring to avoid circular import when importing torch models (#6720) 
Signed-off-by: Rakib Hasan <rhasan@nvidia.com>
 2025-08-11 18:00:42 -04:00
bhsueh_NV
83dbc6c75d
[TRTLLM-5532][feat] store the block of context request into kv cache (#6683) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-08-11 16:14:52 +08:00
Tracin
49bcaa4e95
Add gpt-oss GSM8K test. (#6732) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
 2025-08-10 22:45:43 -04:00
Zero Zeng
4b4b91ab51
[None][feat] improve dataloading for benchmark_dataset by using batch… (#6548) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-11 09:50:41 +08:00
Yechan Kim
60073a7ad9
[None][feat] Support SharedTensor on MultimodalParams (#6254) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-10 17:48:24 -07:00
shaharmor98
b6baa9ed9b
[TRTLLM-6823][doc] Add checkpoint refactor docs (#6592) 
Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>
 2025-08-10 19:47:39 -04:00
shaharmor98
14b36e07d7
[TRTLLM-6174][feat] Enable FP32 mamba ssm cache (#6574) 
Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>
 2025-08-10 16:27:51 -04:00
Gal Hubara-Agam
3c5aec19c2
[#5048][enhance] AutoDeploy: Optimize prepare_inputs (#6634) 
Optimize prepare_inputs routine in AutoDeploy, as part of the effort to reduce the performance gap compared to the default backend.
This PR includes two major fixes, and some other minor tweaks:
1. Avoid back and forth data copies
2. Optimize position ids update by separating the implementation for generation mode and context mode.

Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com>
Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
 2025-08-10 13:55:04 +03:00
Ziyi Xiong
de472828b9
[TRTLLM-6637][feat] Resolve KV cache divergence issue (#6628) 
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
 2025-08-09 23:15:04 +08:00
Yilin Fan
d643aef73c
[Perf] Improve Llama4 performance for small max_seqlen cases (#6306) 
Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>
 2025-08-09 02:58:31 -04:00
Ye Zhang
bcf5ec0c9a
[None][feat] Core Metrics Implementation (#5785) 
Signed-off-by: Ye Zhang <zhysishu@gmail.com>
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-08-09 02:48:53 -04:00
Yibin Li
97787883c3
[TRTLLM-6420][feat] add support for Eclairv2 model - cherry-pick changes and minor fix (#6493) 
Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>
 2025-08-08 21:40:48 -04:00
dongfengy
d06675071e
[None][fix] WAR GPT OSS on H20 with Triton MOE (#6721) 
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
 2025-08-08 19:47:09 -04:00
Mike Iovine
90145cf557
[None][feat] Optimize CUDA graph memory usage for spec decode cases (#6718) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-08-08 13:56:53 -04:00
Wanli Jiang
d45236b253
[TRTLLM-6308][feat] Support Aggregate mode for phi4-mm (#6184) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-08-08 20:09:26 +08:00
Stefan Niebler
b8f036f264
[TRTLLM-6650][fix] Enhance CUDA graph + Beam search to correctly handle padding (#6665) 
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
 2025-08-08 14:00:33 +02:00
Liao Lanyu
32ad7f3c12
[None][fix] Remove lock related typo in py_executor (#6653) 
Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>
 2025-08-08 17:48:57 +08:00
JunyiXu-nv
5f45227a93
[https://nvbugs/5437106][fix] Fix llama4 scout TRTLLM attn_backend (#6690) 
Signed-off-by: Junyi Xu <junyix@nvidia.com>
 2025-08-08 17:48:23 +08:00
Yuxian Qiu
9ff4e75f14
[None][refactor] Combine resmooth_to_fp8_e8m0 and transform_sf_into_required_layout (#6654) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-08-08 17:11:41 +08:00
Li Min
d913955952
[TRTLLM-6898][feat] make fused_moe_cute_dsl work on blackwell (#6616) 
Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>
 2025-08-08 15:03:48 +08:00
2ez4bz
064eb7a70f
[TRTLLM-5252][fix] Propagate mapping to intermediate layers (#6611) 
This commit propagates the mapping to intermediate layers to enable
tensor parallelism (amongst other things) in them.

It also fixes issues with a unit test for TP for pixtral, and adds it to a
test list.

Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
 2025-08-08 01:50:36 -04:00
Enwei Zhu
aee828d98a
[TRTLLM-6854][feat] Enable guided decoding with disagg serving (#6704) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-08-08 12:10:36 +08:00
zhanghaotong
1cf669496a
[None][fix] Fix unnecessary GPU synchronization in torch sampler caused by incorrect tensor reference (#6626) 
Signed-off-by: 皓聪 <zhanghaotong.zht@alibaba-inc.com>
Co-authored-by: 皓聪 <zhanghaotong.zht@alibaba-inc.com>
 2025-08-07 23:44:47 -04:00
NVJiangShao
2f2f5cc72c
[TRTLLM-6744][feat] Remove input_sf swizzle for module WideEPMoE (#6231) 
Signed-off-by: Jiang Shao <91270701+StudyingShao@users.noreply.github.com>
 2025-08-08 11:13:42 +08:00
Daniel Cámpora
efca359b66
[TRTLLM-6785][feat] BREAKING CHANGE Enable TRTLLM sampler by default (#6216) 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
 2025-08-07 22:19:37 -04:00
Iman Tabrizian
82276167e6
[None][feat] Add NCCL Symmetric Integration for All Reduce (#4500) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-08-07 17:28:14 -07:00
Haohang Huang
980929e1a9
[https://nvbugs/5410687][fix] Hopper w4a8 groupwise MoE interleave (#6708) 
Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
 2025-08-07 15:30:16 -07:00
Yuan Tong
db8dc97b7b
[None][fix] Migrate to new cuda binding package name (#6700) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-08-07 16:29:55 -04:00
Mike Iovine
e968f98b43
[None][feat] Clean up ngram auto mode, add max_concurrency to configs (#6676) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-08-07 12:51:47 -04:00
Emma Qiao
3c44b44e45
[None][infra] Fix guardwords (#6711) 
Signed-off-by: qqiao <qqiao@nvidia.com>
 2025-08-07 21:06:47 +08:00
pcastonguay
453a06e6ab
[TRTLLM-6881][feat] Include attention dp rank info with KV cache events (#6563) 
Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
 2025-08-07 14:17:07 +02:00
Enwei Zhu
1b9781e8e7
[TRTLLM-6409][feat] Enable guided decoding with speculative decoding (part 1: two-model engine) (#6300) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-08-07 05:53:48 -04:00
hlu1
8207d5fd39
[None] [feat] Add model gpt-oss (#6645) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>
 2025-08-07 03:04:18 -04:00
amitz-nv
85af62184b
[TRTLLM-6683][feat] Support LoRA reload CPU cache evicted adapter (#6510) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-08-07 09:05:36 +03:00
Yiqing Yan
5fa1914cab
[None][chore] Bump version to 1.1.0rc0 (#6651) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-08-07 13:39:49 +08:00
Izzy Putterman
7e0158b583
Qwen3: Fix eagle hidden states (#6199) 
Signed-off-by: Izzy Putterman <iputterman@nvidia.com>
 2025-08-06 17:05:18 -04:00
Hanjun Cho
80f918cc22
[None][feat] Add Qwen3 MoE support to TensorRT backend (#6470) 
Signed-off-by: gkswns0531 <gkswns0531@gmail.com>
Signed-off-by: hanjuncho <gkswns0531@gmail.com>
Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>
 2025-08-06 17:02:35 +08:00
Zongfei Jing
0ff8df95b7
[https://nvbugs/5433581][fix] DeepGEMM installation on SBSA (#6588) 
Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
 2025-08-06 16:44:21 +08:00
Netanel Haber
83ee91e17b
[None][fix] Fix 6522 mpi.pkl5.intracomm.Request has wait not Wait (#6646) 
Signed-off-by: Netanel Haber <nhaber@nvidia.com>
 2025-08-06 14:18:09 +08:00
JunyiXu-nv
13e0214fe0
[TRTLLM-6263][feat] Enable fp8 SwiGLU to minimize host overhead (#6540) 
Signed-off-by: Junyi Xu <junyix@nvidia.com>
 2025-08-06 10:42:19 +08:00
brb-nv
9a01934dbf
[None][feat] Switch to internal version of MMProjector in Gemma3 (#6572) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-08-05 21:48:23 -04:00
yunruis
3ff4f503ad
[None][opt] ADP schedule balance optimization (#6061) 
Signed-off-by: yunruis <205571022+yunruis@users.noreply.github.com>
 2025-08-06 09:38:02 +08:00
Yechan Kim
c17f4984e2
[None][feat] Refactor Llava-Next (#6478) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-05 17:53:53 -07:00
Aurelien Chartier
6da95f29a9
[None][feat] Add support for fused gate_up_proj scales for FP8 blockwise (#6496) 
Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
 2025-08-05 11:22:32 -07:00
Wanli Jiang
46df8712c8
[https://nvbugs/5355007][fix] Set enable_chunked_context as True by default in trtllm bench (#6582) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-08-05 11:11:36 -07:00
ixlmar
1ebceb790d
[TRTLLM-5508][feat] check input tokens + improve error handling (#5170) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-08-05 18:27:43 +01:00
liji-nv
dcbfa7e509
[https://nvbugs/5252313][fix] Fix torch compile + MTP (#6554) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-08-05 10:31:29 -04:00
Venky
61da2daeb4
[TRTLLM-6761][refactor] Replace LogitBiasLogitsProcessor with embedding bias tensor system (#6464) 
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
 2025-08-05 07:14:24 -07:00
Pengbo Wang @ NVIDIA
c289880afb
[None][fix] fix kimi k2 serving and add test for Kimi-K2 (#6589) 
Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>
 2025-08-05 18:05:33 +08:00
amitz-nv
dc84695520
[TRTLLM-6826][feat] Allow sending more than 2GiB through MPI by using mpi4py.util.pkl5 (#6522) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-08-05 11:28:26 +03:00
danielafrimi
ed801ff74b
[None][fix] Remove expand configuration from mamba2 mixer (#6521) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
 2025-08-05 04:18:25 -04:00
Haohang Huang
c9eebcb454
[TRTLLM-6674][feat] (Breaking Change) Hopper SWA non-cyclic kernels + KV reuse + Spec Dec (#6379) 
Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
Signed-off-by: symphonylyh <31998628+symphonylyh@users.noreply.github.com>
 2025-08-05 07:47:41 +00:00
kris1025
6a3a921284
[TRTLLM-6685][feat] Add speculative metrics for trt llm bench (#6476) 
Signed-off-by: linquanh <linquanh@nvidia.com>
 2025-08-04 15:22:57 -07:00
Olya Kozlova
13cc1c4878
[TRTLLM-5271][feat] best_of/n for pytorch workflow (#5997) 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
 2025-08-04 14:08:06 +02:00
brb-nv
87e4e9f468
[None][chore] Add unit test for Gemma3 lora (#6560) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-08-04 04:56:57 -04:00
Yiqing Yan
3916dbd98b
[None][chore] Bump version to 1.0.0rc6 (#6597) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-08-04 04:39:15 -04:00
Pengyun Lin
a15e33351d
[None][fix] Revert commit 48ddc3d & add test for disagg server with different max_num_tokens (#6259) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-08-04 15:09:51 +08:00
Yuan Tong
a2f271c8e0
[TRTLLM-4406][feat] LLM sleep & wakeup Part 1: virtual device memory (#5034) 
Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-08-04 13:51:01 +08:00
Yechan Kim
ee6ab5be96
chore: add EXAONE4 accuracy test (#6397) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-08-04 10:14:16 +08:00
Jinyang Yuan
df90202b51
[fix] Fix DeepSeek w4a8 weight loading (#6498) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-08-04 10:12:06 +08:00
Chuang Zhu
542f552d0b
use cudaSetDevice to create context ,fix nvbug 5394497 (#6403) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-08-03 13:32:55 -04:00
Shunkangz
67a3fd858b
[None][feat] Add support of scheduling attention dp request (#6246) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
Co-authored-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
 2025-08-01 20:38:01 -04:00
Richard Huo
31802de0b0
[None][fix] Serialize the window_size in the kv event (#6526) 
Signed-off-by: richardhuo-nv <rihuo@nvidia.com>
 2025-08-01 15:25:18 -07:00
Lucas Liebenwein
5247df6ae2
[AutoDeploy] merge feat/ad-2025-07-22 (#6520) 
Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com>
Signed-off-by: Gal Agam <ghubaraagam@cw-dfw-cs-001-login-01.cm.cluster>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: haoguo <67671475+h-guo18@users.noreply.github.com>
Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>
Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>
Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>
Co-authored-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com>
Co-authored-by: Gal Agam <ghubaraagam@cw-dfw-h100-004-328-012.cm.cluster>
Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
Co-authored-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>
Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Co-authored-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>
 2025-08-01 08:51:08 -07:00
brb-nv
7447d6ed85
[TRTLLM-6657][feat] Add LoRA support for Gemma3 (#6371) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-08-01 09:19:54 -04:00
liji-nv
1daa8c3232
[https://nvbugs/5340941][https://nvbugs/5375785] - fix: Wrap attentio… (#6355) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-08-01 07:38:06 -04:00
Yukun He
90856bf97d
[https://nvbugs/5419069][fix] Fix the mismatched layer name components. (#6417) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-08-01 16:32:39 +08:00
Zero Zeng
48768fd720
fix: Fix missing key (#6471) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-08-01 14:25:58 +08:00
Robin Kobus
d3c14682f0
refactor: Remove unused buffers and bindings from sampler (#6484) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-08-01 00:43:03 -04:00
Jaedeok Kim
fbee279909
fix: remove duplicate layer multiplication in KV cache size calculation (#6481) 
Signed-off-by: Jaedeok Kim <jaedeokk@nvidia.com>
 2025-07-31 22:34:34 -04:00
Zongfei Jing
7bb0a78631
Deepseek R1 FP8 Support on Blackwell (#6486) 
Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
Co-authored-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-08-01 10:26:28 +08:00
Venky
8c165fd27a
[TRTLLM-6611][feat] Add warnings and stricter validation to LoraManager adapter loading (#6453) 
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
 2025-07-31 22:22:51 -04:00
Yukun He
00059de380
chore: Improve the AutoTuner log information. (#6368) 
* Change the fallback alert from DEBUG to WARNING level and only do it once.
* Add debug information for profiling cache right after the warmup phase.
* Change the level of exception message during tactic profiling from ERROR to WARNING level. All exception details are pushed to the DEBUG level.
* Other trivial refinements and cleanups.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-08-01 09:19:52 +08:00
brb-nv
2eca0d5925
fix: Fix poor generation with FP8 Gemma3 1B checkpoint (#6499) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-31 17:18:23 -07:00
Simeng Liu
8cf3faa26a
[feat] Auto-enable ngram with concurrency <= 32. (#6232) 
Signed-off-by: Simeng Liu <simengl@nvidia.com>
Signed-off-by: Mike Iovine <miovine@nvidia.com>
Signed-off-by: Mike Iovine <mike.iovine7@gmail.com>
Co-authored-by: Mike Iovine <miovine@nvidia.com>
Co-authored-by: Mike Iovine <mike.iovine7@gmail.com>
 2025-07-31 18:45:51 -04:00
Ziyi Xiong
8062e0fe7c
[TRTLLM-6392][feat] Support turning on/off spec decoding dynamically (#6363) 
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
 2025-07-31 15:31:39 -04:00
shaharmor98
0c42f54a39
Bugfix/fix nemotron nas lora support (#6380) 
Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>
 2025-07-31 13:39:35 -04:00
amitz-nv
1ee7a08d2b
[5830][feat] Improve LoRA cache memory control (#6220) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-07-31 09:26:38 +03:00
dongjiyingdjy
17e0d0fb1a
fix: fix illeagel memory access (#6437) 
Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
 2025-07-31 10:01:34 +08:00
Enwei Zhu
4b299cb77e
feat: Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 (#6408) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-07-31 09:53:52 +08:00
Vadim Gimpelson
25cd4f215e
[PERF] Move calculation Qwen2-VL's rotary_cos_sin to LLM worker process (#6004) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@centml.ai>
 2025-07-31 09:35:24 +09:00
shaharmor98
f9cf683e39
add propagation of trust_remote_code to OpenAIServer (#6446) 
Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>
 2025-07-30 15:25:41 -04:00
Wanli Jiang
9632dba02e
feat: TRTLLM-6450 update long rope for phi3.5/phi4-mini/phi4-mm (#6353) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-07-30 09:20:16 -07:00
NVShreyas
e67f4da9b5
[Perf]: Add residual, norm for nemotron_nas models (#6455) 
Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>
 2025-07-30 09:10:38 -07:00
Chang Liu
b4065d8ca6
[TRTLLM-6654][feat] Add support for external multimodal embeddings (#6263) 
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
 2025-07-30 10:00:15 -04:00
pcastonguay
e7ae5e2824
feat: Add support for disaggregation with pp with pytorch backend (#6369) 
Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>
Signed-off-by: raayandhar <rdhar@nvidia.com>
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
Signed-off-by: pcastonguay <55748270+pcastonguay@users.noreply.github.com>
Co-authored-by: raayandhar <rdhar@nvidia.com>
Co-authored-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-07-30 09:42:13 -04:00
tomeras91
a2514d93fc
[nvbug 5380101][fix] Fix nemotronNAS loading for TP>1 (#6447) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
 2025-07-30 07:22:32 -04:00
QI JUN
2fe9cc0889
chore: remove draft_model_engine from init parameter list of PyExecutor (#6325) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-30 03:31:49 -04:00
QI JUN
1f39a11af0
chore: clean code of PyExecutor (#6445) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-30 02:11:43 -04:00
2ez4bz
d6eed1b624
[fix] Switch placement of image placeholder for mistral 3.1 (#6435) 
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
 2025-07-30 14:10:36 +08:00
Jinyang Yuan
a427f5bece
[fix] Fix wide EP when using DeepEP with online EPLB (#6429) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-07-30 00:13:18 -04:00
Zheng Duan
c9ed1ab436
[TRTLLM-6549] chore: record delay introduced by disaggregated serving in kv cache measure (#6135) 
Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>
 2025-07-30 10:39:40 +08:00
peaceh-nv
5b420ad267
Rename layer to comply with deepseek (#6393) 
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
 2025-07-30 10:00:48 +08:00
Yechan Kim
d6eb8e2366
fix: support mixture of text & multimodal prompts (#6345) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-07-30 08:52:31 +08:00
Yunfan Fan
1a8e28d295
[FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine (#6344) 
Signed-off-by: fanyunfan <2569548856@qq.com>
Co-authored-by: fanyunfan <2569658856@qq.com>
 2025-07-30 07:13:44 +08:00
Yan Chunwei
ad662ddcdd
chore: disallow arbitrary in llm_args.Configs (#6367) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-07-29 16:16:52 -04:00
Michal Guzek
7efe3cb0cd
[fix] Add detokenization-based stop word logic to LLM API (#5948) 
Signed-off-by: moraxu <mguzek@nvidia.com>
Signed-off-by: Michal Guzek <mguzek@nvidia.com>
 2025-07-29 10:16:59 -07:00
Yukun He
0eee2e2850
[5385981] fix: Update the usage of VisionAttention init API. (#6413) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-07-29 16:41:48 +08:00
QI JUN
13e24ab1cb
chore: remove unused code in PyExecutor (#6351) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-29 16:24:26 +08:00
Frank
d2a04abb95
[fix] Fixes to parameter usage and low latency configuration. (#6343) 2025-07-29 01:36:13 -04:00
nv-guomingz
49044733e1
chore: delete useless gitkeep files. (#6400) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-07-28 11:38:30 -04:00
QI JUN
4efc6496b7
chore: add _prepare_and_schedule_batch function in PyExecutor (#6365) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-28 05:50:27 -04:00
Yan Chunwei
45d441e60c
[TRTLLM-5061] chore: add status tags to LLM API reference (#5707) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-07-28 15:57:07 +08:00
Zero Zeng
c9b8b6180f
Add Acceptance Rate calculation to benchmark_serving (#6240) 
Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>
 2025-07-28 14:00:58 +08:00
Jinyang Yuan
97f7e12588
[fix] Fix perf regression caused by MoE autotuner when using DeepEPLowLatency (#6288) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-07-28 01:37:11 -04:00
Chang Liu
dc757799e1
[nvbugs/5401156][fix] Avoid import all models when import trtllm._common (#6266) 2025-07-27 23:29:21 -04:00
Void
f172face98
DeepEP LL dispatch FP4 (#6296) 
Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>
 2025-07-28 11:25:42 +08:00
Yukun He
93a0fd0a23
[TRTLLM-6445] feat: Enable AllReduce-associated fusion patterns in Llama3/4. (#6205) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-07-28 09:36:26 +08:00
YueWeng
2dd3186727
fix: remove cudaStreamSynchronize when using relaxed acceptance (#5262) 
Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>
 2025-07-28 09:18:41 +08:00
Ziyi Xiong
d853811190
[https://nvbugs/5402719][fix]: Add cuda graph dummy requests to the spec_resource_manager (#6258) 
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
 2025-07-26 20:32:39 -04:00
Michal Guzek
08d57123f9
[nvbug/5374773] chore: Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache (#5974) 
Signed-off-by: moraxu <mguzek@nvidia.com>
 2025-07-25 18:10:40 -04:00
ameynaik-hub
1e5e71aa42
Mtp optimizations round1 (#5689) 
Signed-off-by: Amey Naik <212485788+ameynaik-hub@users.noreply.github.com>
Co-authored-by: Kefeng-Duan <176893526+Kefeng-Duan@users.noreply.github.com>
 2025-07-25 13:48:27 -04:00
nv-guomingz
b8d4cb8beb
feat: Support JSON Schema in OpenAI-Compatible API (#6321) 
Signed-off-by: noiji <52301388+noiji@users.noreply.github.com>
 2025-07-25 12:55:56 -04:00
xiaoqi
a0aecf0476
[feat]: support logit_bias (#5354) 
Signed-off-by: xq25478 <xq25478@qq.com>
Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: hexiao.xq <hexiao.xq@antgroup.com>
Co-authored-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>
Co-authored-by: hexiao.xq <hexiao.xq@antgroup.com>
Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-25 09:37:41 +00:00
liji-nv
e07fff4f78
[https://nvbugs/5340941] - fix: Correct custom ops used by Qwen3 Moe … (#6285) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-07-25 14:49:45 +08:00
Mike Iovine
0f2f11f90b
[TRTLLM-6453][feat] Support chunked prefill on spec decode 2 model (#6104) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-07-24 21:50:11 -04:00
Linda
9a99e6d6d7
fix: integration tests with nanobind (#6326) 
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>
 2025-07-25 09:23:20 +08:00
Shiyu Li
375f74ecb2
[fix][nvbugs/5399355] Fix Lamport buffer clear issue for MNNVL TwoShot Allreduce and add FP16 support. (#6237) 
Signed-off-by: Shiyu Li <shili@nvidia.com>
 2025-07-25 08:01:40 +08:00
Frank
f8f5ba65fc
[fix] Update to remove popping of KV cache and other args. (#6310) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-07-24 15:54:33 -04:00
Stefan Niebler
0df758ec9f
[TRTLLM-6650][feat] Enhance beam search support with CUDA graph integration (#6217) 
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
 2025-07-24 18:04:41 +02:00
bhsueh_NV
7b6aadc800
[Fix][nvbug 5401163][nvbug 5404726][Qwen3] Fix bug of MoE on tp > 1 with trtllm moe backend (#6235) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-07-24 21:47:37 +08:00
liji-nv
14d94a3856
feat: Add non UB AR + Residual + Norm + Quant fusion (#6320) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-07-24 05:51:43 -04:00
Lizhi Zhou
a63a1ac7f9
[TRTLLM-6444] Add some UCX trouble shooting docs and print UCX related logs (#6085) 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
 2025-07-24 16:21:01 +08:00
QI JUN
428e34080f
chore: remove unused variables in pyexecutor (#6280) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-24 13:16:15 +08:00
Stefan Niebler
2486eb778e
[TRTLLM-6651][feat] Enable Overlap scheduler + Beam Search in TRTLLM Sampler (#6223) 
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
 2025-07-23 12:30:50 +02:00
YueWeng
ed62a06eef
[nvbug/5322354] fix PD + MTP + overlap scheduler accuracy issue (#6136) 
Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>
 2025-07-23 14:53:37 +08:00
QI JUN
a8253b942f
chore: remove duplicate should_stop_processing check (#6242) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-23 14:11:23 +08:00
Yechan Kim
83c3ed128b
chore: set default device to cpu on Multimodal models (#5994) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-07-22 21:45:31 -07:00
Erin
5636c67388
fix: nvbug_5398806 (#6239) 2025-07-23 11:45:11 +08:00
Venky
9538c8d0e5
Add basic Nemo Ckpt Lora Loading in pytorch flow (#6019) 2025-07-22 19:42:45 -07:00
wili
8ecdeee300
[refactor] Simplification of Speculative decoding configs - Part 2 (#5936) 
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-07-23 09:20:27 +08:00
Lucas Liebenwein
41fb8aa8b1
[AutoDeploy] merge feat/ad-2025-07-07 (#6196) 
Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com>
Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>
Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Co-authored-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>
Co-authored-by: Neta Zmora <nzmora@nvidia.com>
Co-authored-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>
Co-authored-by: Frida Hou  <201670829+Fridah-nv@users.noreply.github.com>
Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
Co-authored-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com>
 2025-07-23 05:11:04 +08:00
2ez4bz
ab7434ac62
[feat] Enable TP and batching for PixtralVisionModel / Mistral3VLM (#6152) 
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
 2025-07-22 11:06:41 -07:00
John Calderon
b7c8a672da
[Issue 6193] Fix gemma3vl weight loader (#6233) 
Signed-off-by: John Calderon <johncalesp@gmail.com>
 2025-07-22 10:32:18 -07:00
danielafrimi
ff9963978a
Add register_fake for finegrained_mixed_dtype_gemm torch_op (#6255) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
 2025-07-22 16:59:55 +03:00
Yiqing Yan
3e18ee5fe1
chore: bump version to 1.0.0rc5 (#6252) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-07-22 16:24:28 +08:00
Pengyun Lin
48ddc3d4b9 [fix]: Revert commit 388b491 (#6143) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-22 12:48:00 +08:00
Yi Zhang
eb7d0f84b5 [nvbugs/5368410][fix] Disable moe allreduce for multi node (#5918) 
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
 2025-07-22 12:48:00 +08:00
Fanrong Li
c66941036f fix: fix index out of bounds error in spec decoding (#5954) 2025-07-22 12:48:00 +08:00
Shunkangz
ee45e0c63f
feat: Refactor the fetching request logic (#5786) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-07-22 09:16:28 +08:00
Chang Liu
7381f1dba7
[TRTLLM-5059][feat] Add KV cache reuse support for multimodal models (#5444) 
Only supports qwen in this PR
 2025-07-21 16:11:58 -07:00
Ziyi Xiong
d7f0b0ab68
[fix] Correct the returned value of has_spec_drafter (#6178) 
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
 2025-07-21 11:38:59 -04:00
Pengyun Lin
9832bef07d
[BREAKING CHANGE]: change default backend to PyTorch in trtllm-serve (#5717) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-21 21:09:43 +08:00
liji-nv
3e0fb60e50
[TRTLLM-4279] feat: Multistream initial support for torch compile flow (#5847) 
Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
 2025-07-21 19:10:22 +08:00
Linda
3efad2e58c
feat: nanobind bindings (#6185) 
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>
 2025-07-21 08:56:57 +01:00
Yuening Li
e8c068b4b1
[TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow (#5850) 
Signed-off-by: Yuening Li <62227368+yueningl@users.noreply.github.com>
Co-authored-by: Yuening Li <62227368+yueningl@users.noreply.github.com>
 2025-07-21 15:17:35 +08:00
Jinyang Yuan
88076eecd0
[fix] Fix can_use_alltoall in fused_moe_wide_ep.py (#6173) 
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
 2025-07-21 10:53:07 +08:00
brb-nv
ca9bc5727e
fix: Flush stale PlanParams with custom attention mask (#6163) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-21 09:55:09 +08:00
brb-nv
a433ebad2b
enh: Lift expectation of single image per sample in Gemma3 VLM (#6195) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-21 08:43:07 +08:00
danielafrimi
5300a99bd8
W4A8 GEMM (#6005) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
 2025-07-20 17:34:57 +03:00
amitz-nv
98428f330e
[TRTLLM-5826][feat] Support pytorch LoRA adapter eviction (#5616) 
Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-07-20 08:00:14 +03:00
Void
118307c224
DeepEP LL support variable hidden size and tokens num (#6141) 
Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>
 2025-07-20 09:32:41 +08:00
Pengyun Lin
69e9f6d489
[fix]: Skip prompt length checking for generation only requests (#6146) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-19 21:26:37 +08:00
Ziyi Xiong
66030ef815
[TRTLLM-6452][feat]: Two-model engine KV cache reuse support (#6133) 
Signed-off-by: ziyixiong-nv <fxiong@nvidia.com>
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
 2025-07-19 13:17:15 +08:00
Rashid Kaleem
152e2df43b
[Disaggregated] Add retry knobs and handling (#5808) 
Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>
Signed-off-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>
Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>
 2025-07-19 07:27:59 +08:00
John Calderon
fc8b29c4ff
[Issue 5927][fix] Avoid memory calls during broadcast for single GPU (#6010) 
Signed-off-by: John Calderon <johncalesp@gmail.com>
 2025-07-18 14:21:03 -07:00
Netanel Haber
d9a3530048
[nvbug/5393888][nvbug/5393042] Always use py_seq_slot (#6147) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
 2025-07-18 22:45:16 +03:00
Stefan Niebler
6d7874a467
[nvbugs/5369799] fix: Update disaggregation handling in sampler (#5762) 
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
 2025-07-19 01:40:46 +08:00
xiaoqi
28858c8711
feat(eagle3):support qwen3 dense model (#5879) 
Signed-off-by: xq25478 <xq25478@qq.com>
 2025-07-19 01:24:32 +08:00
Bo Li
07e8813984
feat: Remove padding in attention DP. (#6064) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-07-18 23:30:34 +08:00
Stefan Niebler
fd6ce7f20e
[ci] Speedup beam search unit tests with fixtures for LLM (#5843) 
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
 2025-07-18 22:54:49 +08:00
Erin
9522cde464
fix: NVBug 5385576 py_batch_idx issue (#6153) 
Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
 2025-07-18 22:36:43 +08:00
Robin Kobus
ec2b953e7e
refactor: Enhanced handling of decoder requests and logits within the batch manager (#6055) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-07-18 12:12:08 +02:00
Aurelien Chartier
812243bdd6
feat: add support for Modelopt fp8_pb_wo quantization scheme (#6106) 
Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
 2025-07-18 10:35:12 +08:00
Zhenhuan Chen
992b273045
[https://nvbugs/5387375] fix(scaffolding): fix scaffolding aime test in test_e2e (#6140) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-07-18 10:34:37 +08:00
yifeizhang-c
0155e7a3a1
[TRTLLM-6368] Update deepep dispatch API (#6037) 
Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>
 2025-07-18 10:13:31 +08:00
Iman Tabrizian
b75e53ab69
Revert "feat: nanobind bindings (#5961)" (#6160) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-07-18 10:12:54 +08:00
qixiang-99
2c90203c36
Refactor KVCacheManager: Simplify token availability calculation and … (#6134) 
Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>
 2025-07-17 13:33:33 -07:00
Frank
161490f039
[fix] Fixes KV Cache overrides in trtllm-bench (#6103) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-07-18 03:44:44 +08:00
2ez4bz
8480c120b1
[fix] Fix Mistral3VLM weight-loading & enable in pre-merge (#6105) 
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
 2025-07-17 11:04:17 -07:00
Iman Tabrizian
10dbf4f0f4
[fix] Remove duplicated KVCache transmission check (#6022) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-07-17 12:02:19 -04:00
Linda
5bff317abf
feat: nanobind bindings (#5961) 
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>
 2025-07-17 22:42:52 +08:00
Ziyi Xiong
58d22a72f1
[TRTLLM-6352][feat] Migrate EAGLE3 and draft/target speculation to Drafter (#6007) 
Signed-off-by: ziyixiong-nv <fxiong@nvidia.com>
 2025-07-17 21:15:01 +08:00
Enwei Zhu
21efb50068
[TRTLLM-6406] feat: Enable guided decoding with overlap scheduler (#6000) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-07-17 17:46:10 +08:00
Chuang Zhu
44c70c88f9
chore:[BREAKING CHANGE] use cacheTransceiverConfig as knobs for disagg service (#5234) 
Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>
 2025-07-17 17:42:07 +08:00
Iman Tabrizian
d4d21a106e
[fix] Release slots with spec decode + disagg (#5975) (#6032) 
Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>
Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-07-17 12:58:18 +08:00
Shiyu Li
6e1aee6fd6
[fix] Performance Optimization for MNNVL TwoShot Kernel (#5934) 
Signed-off-by: Shiyu Li <shili@nvidia.com>
Co-authored-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
 2025-07-17 10:49:51 +08:00
chenfeiz0326
fe070a0168
test: Update Llama4 Scout FP4 & FP8 accuracy tests (#5901) 
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
 2025-07-17 09:41:18 +08:00
Wanli Jiang
2d2b8bae32
feat: TRTLLM-5574 Add phi-4-multimodal pytorch-backend support (#5644) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-07-17 06:30:58 +08:00
qixiang-99
e09e409dfb
Fix: Enhance ModelConfig for kv cache size calculations (#5868) 
Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>
 2025-07-16 14:41:31 -07:00
Mike Iovine
fa34cb7234
[refactor] Clean up drafter/resource manager creation logic (#5805) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-07-16 12:45:46 -07:00
shaharmor98
e0836f9ca9
[TRTLLM-5493] Add core infrastructure to enable loading of custom checkpoint formats (#5372) 
Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>
 2025-07-17 00:50:30 +08:00
Wanli Jiang
9354114f68
fix: Update trtllm args issues with extra nested config (#5996) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-07-16 12:41:45 -04:00
Bo Li
fc2347eaf5
chore: Cleanup disable_fp4_allgather. (#6006) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2025-07-16 17:54:36 +08:00
Yan Chunwei
a02606a9e2
[TRTLLM-5530][BREAKING CHANGE] refactor: unify KvCacheConfig in LLM class for pytorch backend (#5752) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-07-16 16:42:59 +08:00
Yan Chunwei
7568deb2f1
[nvbug/5387226] chore: add propogation for trust_remote_code to AutoConfig (#6001) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-07-16 16:05:38 +08:00
Yiqing Yan
e51c541617
chore: Bump version to 1.0.0rc4 (#6086) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-07-16 13:02:23 +08:00
Wanli Jiang
8679a058a3
fix: Unable to load phi4-model with tp_size>1 (#5962) 
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-07-16 11:39:41 +08:00
danielafrimi
edab7532dd
feat/add latency support for trtllm bench (#3730) 
Signed-off-by: Ubuntu <dafrimi@nvidia.com>
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
Signed-off-by: Frank <3429989+FrankD412@users.noreply.github.com>
Co-authored-by: Daniel Afrimi <dafrimi@nvidia.com>
Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com>
 2025-07-15 13:13:49 -07:00
Fanrong Li
7a1af1c738
Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/5947 (#5989) 
Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-07-16 01:33:12 +09:00
Xiaodong (Vincent) Huang
0523f77b36
support TRTLLM_DEEP_EP_TOKEN_LIMIT to allow run deep-ep on memory-con… (#5684) 
Signed-off-by: Vincent Huang <vincenth@nvidia.com>
 2025-07-15 18:34:21 +03:00
Tailing Yuan
4a26bd6500
Fix: pad DeepEP fp4 recv tensors if empty (#6048) 
Signed-off-by: Tailing Yuan <yuantailing@gmail.com>
 2025-07-15 23:14:01 +09:00
MinaHuai
9ebc3ab9c4
[nvbugs/5385972][nvbugs/5387423][Fix] Minor fix for llava_next/llava_onevision (#5998) 
Signed-off-by: Mina Huai <121143971+MinaHuai@users.noreply.github.com>
 2025-07-15 10:01:35 -04:00
Jaedeok Kim
ab1c54709d
fix: adjust window sizes of VSWA at torch backend (#5880) 
Signed-off-by: Jaedeok Kim <jaedeokk@nvidia.com>
 2025-07-15 17:41:54 +08:00
nv-guomingz
4e4d18826f
chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… (#6003) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-07-15 15:50:03 +09:00
Lucas Liebenwein
e499f6c44a
[Fix] check for ImportError or ModuleNotFoundError for deep_ep_utils (#6026) 
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
 2025-07-15 14:31:35 +09:00
Rashid Kaleem
2ea4077993
[Model load] Fix llama min-latency model load (#5883) 
Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>
 2025-07-15 09:29:19 +08:00
ixlmar
f225f5cd2e
[nvbugs-5318143] fix: restrict PyTorch memory usage to avoid OOMs (#5964) 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
 2025-07-15 06:49:42 +08:00
brb-nv
f5f5be9e94
enh: Bidirectional mask with multiple images for Gemma3 (#5976) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-14 22:39:18 +08:00
brb-nv
1a2d96919c
feat: Update Gemma3 Vision Encoder (#5973) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-14 22:38:10 +08:00
Yechan Kim
63139fdcff
feat: EXAONE4.0 support (#5696) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-07-14 22:28:10 +09:00
Zhenhuan Chen
30608a5e6d [https://nvbugs/5355316] fix: update torch.compile option to fix triton store_cubin error (#5865) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-07-14 17:17:30 +08:00
Robin Kobus
5a61d64b5b [nvbugs/5345391] fix: chunked prefill + overlap scheduling (#5761) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-07-14 17:17:30 +08:00
Pengyun Lin
388b4919b8 [nvbug 5304752][fix] enhance _check_arguments to filter illegal requests for pytorch backend (#5541) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-14 17:17:30 +08:00
Pengyun Lin
6992616c1f [nvbug 5004744][fix] rewrite completion API to avoid repetitive tokens (#5201) 
Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>
 2025-07-14 17:17:30 +08:00
Iman Tabrizian
c8874a7f94 [nvbug/5337601][fix] Fix disagg + speculative decoding (#5558) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-07-14 17:17:30 +08:00
Dom Brown
afaa388bee [TRTLLM-6100] fix: Nvbug 5356427: autotuned TRTLLM Gen fp8 block scale MoE illegal memory access (#5676) 
Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-07-14 17:17:30 +08:00
WeiHaocheng
4d8920982a
fix: set allreduce strategy to model config (#5955) 
Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
 2025-07-14 17:59:11 +09:00
dominicshanshan
c9e7f831dc
Breaking change: perf: [TRTLLM-4662] Enable cuda graph by default (#5480) 
Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
 2025-07-14 16:42:23 +08:00
wili
cfcb97af0e
[BUG5388075][fix] Fix error in post-merge-tests (#5949) 
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-07-14 14:33:39 +09:00
QI JUN
ce39409530
fix cancel request logic (#5800) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-07-14 10:23:20 +08:00
Mike Iovine
8950223f6f
[fix] Remove SpecConfig and fix thread leak issues (#5931) 
Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>
 2025-07-12 21:03:24 +09:00
Enwei Zhu
bc1d4fb5da
[NvBug 5378370] fix: Fix alltoall for llama4 (apply_router_weight_on_input=True) (#5902) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-07-12 15:50:31 +09:00
Thor Johnsen
041f1fa513
[TRTLLM-6264] Fix flaky test_e2e.py::test_openai_lora (#5885) 
Signed-off-by: thorjohnsen <41591019+thorjohnsen@users.noreply.github.com>
 2025-07-11 16:20:41 -07:00
2ez4bz
6304866ce8
[refactor] Move vision parts from processor to model for Gemma3 (#5888) 
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
 2025-07-11 15:13:51 -07:00
brb-nv
0385f89abc
test: Fix Gemma3 unit tests due to transformers upgrade (#5921) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-10 17:24:10 -07:00
Void
854655f2f7
deepEP fp4 post quant all2all dispatch (#5881) 
Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>
 2025-07-11 08:18:54 +08:00
Frank
aa4eebe973
[enhance] Add the ability to write a request timeline. (#5258) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
Signed-off-by: Frank <3429989+FrankD412@users.noreply.github.com>
 2025-07-10 17:15:30 -07:00
wili
2e3cf42e03
[refactor] Simplification of Speculative decoding configs (#5639) 
Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com>
Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>
 2025-07-10 11:37:30 -04:00
Kaiyu Xie
7b09a415c1
fix: Make the bench serving script compatible with different usages (#5905) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-07-10 19:36:26 +08:00
Enwei Zhu
055c4a9fe6
[NvBug 5370718, 5371538] fix: Fix incremental detokenization (#5825) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-07-10 16:30:00 +08:00
CarstyYou
dc32f9ae73
[fix] fix tileN cannot % 16==0 & support sm89 deepgemm bmm (#5531) 
Signed-off-by: CarstyYou <186021327+CarstyYou@users.noreply.github.com>
 2025-07-10 15:16:18 +08:00
Anthony Chang
7d21b55b5a
[feat] Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE (#5723) 
Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>
 2025-07-10 14:06:50 +08:00
Yan Chunwei
07f6da763d
[TRTLLM-5530] chore: rename LLM.autotuner_enabled to enable_autotuner (#5876) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-07-10 11:31:35 +08:00
Hanjun Cho
6490a27ad7
[feat] Add TensorRT-Engine Qwen3 (dense) model support (#5650) 
Signed-off-by: Ubuntu <ubuntu@ip-10-0-20-146.us-west-2.compute.internal>
Signed-off-by: Hanjun Cho <46752251+gkswns0531@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-10-0-20-146.us-west-2.compute.internal>
 2025-07-10 10:26:06 +08:00
brb-nv
3209b31665
feat: Custom masking utils for Gemma3 VLM (#5853) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-10 06:18:04 +09:00
2ez4bz
87fe44fd29
feat(models): Mistral3.1 VLM pytorch backend support (#5529) 
Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>
 2025-07-09 13:17:40 -07:00
Chang Liu
b61a717275
[1/N][TRTLLM-5195][feat] Share PyTorch tensor between processes (#5396) 2025-07-10 05:12:53 +09:00
Wanli Jiang
3f7cedec7c
Update transformers to 4.53.0 (#5747) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
 2025-07-09 09:32:24 -07:00
DylanChen-NV
74dca0aa7b
[NVBUG-5304516/5319741]Qwen2.5VL FP8 support (#5029) 
Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
 2025-07-09 23:16:42 +08:00
tomeras91
5aa958a11a
[TRTLLM-5838][fix] fix max batch size and max tokens in kv cache estimations for Nemotron-H (#5371) 
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
 2025-07-09 11:30:15 +03:00
Dom Brown
3e3b1769ad
[TRTLLM-5881] feat: Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner (#5764) 
Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-07-09 08:21:58 +01:00
dongxuy04
dd3c736c7e
chore: some refactor on WideEP (#5727) 
Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
 2025-07-09 14:26:57 +08:00
chenfeiz0326
64fd64fcf2
[TRTLLM-6262] Fix Llama4 Scout FP4 crash issue (#5834) 
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
 2025-07-09 14:23:21 +08:00
Chang Liu
4df5f96c8d
[Bugfix] LLama4: fix for llama4 multimodal support (#5809) 2025-07-09 13:03:40 +09:00
Xianjie Qiao
5ab1cf5ae6
Remove unnecessary benchmarking results (#5852) 
Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
 2025-07-09 11:19:06 +08:00
brb-nv
2bd09ed2d4
fix: Skip rope scaling for local layers in Gemma3 VLM (#5857) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-09 10:10:33 +08:00
Omer Ullman Argov
d6d2ab2c99
[fix] Catch inference failures in trtllm-bench (#5841) 
Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>
 2025-07-09 03:53:03 +03:00
Iman Tabrizian
c508b994b6
Fix lost requests for disaggregated serving (#5815) 
Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>
 2025-07-09 08:42:45 +09:00
Kaiyu Xie
bb5b16fcb9
feat: Return context response immediately when stream_interval > 1 (#5836) 
Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
 2025-07-09 00:19:57 +09:00
Raayan Dhar
e3268a4221
[TRTLLM-5847][feat] Support n-gram speculative decoding with disagg (#5732) 
Signed-off-by: raayandhar <rdhar@nvidia.com>
 2025-07-08 09:39:58 -04:00
Yukun He
e104f8bbb5
[5305318] fix: Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. (#5801) 
Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-07-08 19:51:05 +08:00
Yegor
b01d1c28f7
[feat] Detokenize option in /v1/completions request (#5382) 
Signed-off-by: Yegor <75512761+Wokzy@users.noreply.github.com>
Signed-off-by: Yegor Yershov <yegor6741@gmail.com>
 2025-07-08 19:36:04 +08:00
xiweny
eaf8bec88b
fix: Disaggregate serving with attention DP (#4993) 
Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>
 2025-07-08 16:15:03 +08:00
Yiqing Yan
5203a0f6df
chore: bump version to 1.0.0rc3 (#5819) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-07-08 16:04:40 +09:00
Zhenhuan Chen
dee6644ed9
feat(scaffolding): add streaming scaffolding_llm.generate_async support (#5345) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-07-08 15:08:40 +09:00
nv-guomingz
0be41b6524
Revert "chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie…" (#5818) 2025-07-08 13:15:30 +09:00
Yechan Kim
5bc3a15f10
feat: add MultimodalParams & putting all multimodal params into it and refactor HyperCLOVAX & Qwen2/2.5-VL (#5522) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-07-07 18:03:12 -07:00
nv-guomingz
5a8173c121
chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… (#5795) 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
 2025-07-08 08:52:36 +08:00
Robin Kobus
30a19fcf7c
[TRTLLM-6291] feat: Add user-provided speculative decoding support (#5204) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-07-07 16:30:43 +02:00
Tailing Yuan
85b4a6808d
Refactor: move DeepEP from Docker images to wheel building (#5534) 
Signed-off-by: Tailing Yuan <yuantailing@gmail.com>
 2025-07-07 22:57:03 +09:00
Daniel Cámpora
1260e2f33f
feat: Optimize TRTLLM Sampler perf single beam single step (#5550) 
Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>
 2025-07-07 15:44:47 +02:00
DylanChen-NV
5ca2b9bb15
[TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow (#5615) 
Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
 2025-07-07 18:04:57 +08:00
Yan Chunwei
dfce61f4b9
[TRTLLM-5530][BREAKING CHANGE] refactor: LLM arglist rename mixed_sampler to enable_mixed_sampler (#5751) 
Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-07-07 17:05:14 +08:00
Zheng Duan
de10774c2e
chore: log stack trace on error in openai server (#5749) 
Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>
 2025-07-07 14:54:36 +08:00
Daniel Stokes
ec6c7dff1a
feat: Add support for MXFP8xMXFP4 in pytorch (#5535) 
Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>
 2025-07-06 15:32:06 -07:00
Robin Kobus
ae27261094
refactor: decoding inputs (#5679) 
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
 2025-07-06 08:21:02 +02:00
Xianjie Qiao
b1976c2add
Add wide-ep benchmarking scripts (#5760) 
Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
Signed-off-by: Xianjie Qiao <5410381+qiaoxj07@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-05 19:29:39 +08:00
Xianjie Qiao
089fd55eda
Add dummy all_reduce for kernel breakdown (#5745) 
Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>
 2025-07-05 13:08:58 +09:00
Frank
d61893dc77
[fix] Update to properly set cuda graphs in trtllm-bench overrides. (#5634) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-07-05 05:19:16 +09:00
Stefan Niebler
d1112aac37
[TRTLLM-3442] feat: added beam search support to the PyTorch Workflow (#5333) 
Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>
 2025-07-05 01:35:13 +09:00
HuiGao-NV
3ed3bbcb5d
Fix: pass allreduce strategy to pytorchConfig (#5746) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-07-04 21:32:13 +09:00
Shunkangz
32339d1b20
Raise shut down error for each request (#4936) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-07-04 18:58:24 +09:00
Tailing Yuan
e134a52e07
Perf: reduce DeepEPLowLatency memory and time (#5712) 
Signed-off-by: Tailing Yuan <yuantailing@gmail.com>
 2025-07-04 14:46:28 +08:00
Shunkangz
a79d8c9f5e
Fix none response in PD (#5422) 
Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-07-04 14:25:10 +08:00
brb-nv
cdaa6abce7 fix: Investigate Gemma3 1B decoder output discrepancy (#5564) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-07-04 13:14:13 +08:00
Frank
819ae903de [https://nvbugspro.nvidia.com/bug/5351333][fix] Update to chunking calculation. (#5625) 
Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
 2025-07-04 13:14:13 +08:00
Clay
7a319524da
feat: support more parameters in openai worker of scaffolding (#5115) 
Signed-off-by: Clay <ccs96307@gmail.com>
 2025-07-04 09:35:34 +08:00
Lucas Liebenwein
24ac9b5f69
[AutoDeploy] merge feat/ad-2025-06-29 (#5737) 
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>
Co-authored-by: Neta Zmora <nzmora@nvidia.com>
Co-authored-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>
 2025-07-04 10:21:18 +09:00
Netanel Haber
aa72d39b72
MTP and derivatives: Align sample state with trtllm sampler sample state (#5675) 
This PR moves MTPSampler and derivatives to use the universal seq_slot indexing for sampling.
This is the last piece of the puzzle: After this, all of the samplers will use this format.
See: 6ee94c7
Signed-off-by: Netanel Haber <nhaber@nvidia.com>
 2025-07-03 19:55:48 +02:00
Zhenhuan Chen
528ff52ef4
[https://nvbugs/5365714] fix(scaffolding): use default LLM rather than trt backend LLM (#5705) 
Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>
 2025-07-03 23:54:20 +09:00
Rashid Kaleem
2b0c87e613
[ModelLoad] Concurrent load model (#5291) 
Signed-off-by: Rashid K <rkaleem@nvidia.com>
Co-authored-by: Zhihan Jiang <68881590+nvzhihanj@users.noreply.github.com>
 2025-07-03 22:18:04 +08:00