TensorRT-LLMs

mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-02-06 03:01:50 +08:00

Author	SHA1	Message	Date
Chang Liu	726ac07cc0	[https://nvbugs/5549081 ][fix] Fix device id assignment for some vision models (#8070 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>	2025-10-01 23:28:05 -04:00
brb-nv	bd3d0ad233	[TRTLLM-7733][feat] Executor changes to support helix parallelism (#7972 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-10-01 22:13:03 -04:00
Izzy Putterman	1ad7bc4c78	[None][feat] Draft: Save state first pass (#7012 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-10-01 18:40:55 -04:00
Frida Hou	de99e23696	[#5860 ][feat] Add ModelOPT INT4 awq fake quant support in AutoDeploy (#7770 ) Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>	2025-10-01 13:13:45 -07:00
Yibin Li	d7581bb551	[TRTLLM-8031][feat] Add chunked return_generation_logits logic (#7831 ) Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>	2025-10-01 12:47:07 -04:00
Grzegorz Kwasniewski	6fd225833c	[TRTLLM-6342][bug] Fix shape propagation after TP sharding (#7912 ) Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>	2025-10-01 11:15:46 -04:00
sychen52	ba8abeab10	[OMNIML-2336][feat] add W4A8 NVFP4 FP8 fused moe (#7968 ) Signed-off-by: Shiyang Chen <shiychen@nvidia.com>	2025-10-01 02:39:33 -04:00
peaceh-nv	808e556c79	[None][fix] : Fix OOM issue when dp padding is enabled (#8052 ) Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>	2025-10-01 09:10:00 +08:00
brb-nv	84aa3c981e	[None][chore] Waive failing MNNVL alltoall multi-gpu test (#8106 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-09-30 20:05:42 -04:00
mpikulski	ee5ae49337	[TRTLLM-8269][fix] Revert "do not explicitly pass temperature=0 to select greedy sampling" (#8103 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-09-30 16:53:49 -04:00
Guoming Zhang	b4be0d2e4c	[None][chore] Refine qwen3-next implementation. (#8064 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-09-30 15:05:13 -04:00
Yiqing Yan	1560cca227	[None][chore] Bump version to 1.2.0rc1 (#8097 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-09-30 06:00:25 -04:00
Yechan Kim	948b8b9569	[None][fix] Fix CUDA graph for Qwen2.5-VL (#8047 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-09-30 14:40:03 +08:00
Kaiyu Xie	b0cb9ca50e	[None] [test] Add MNNVL AlltoAll tests to pre-merge (#7466 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> Co-authored-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-09-29 23:12:24 -04:00
Lucas Liebenwein	dcfd3ef81c	[#4593 ][feat] AutoDeploy: Linear Attention Support (SSM + causal_conv + Bamba + Nemotron-H) (#8068 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Co-authored-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-09-29 22:41:06 -04:00
Cao Dong	62010c0ab7	[None][feat] Return topk logprobs in torch backend (#7976 ) Signed-off-by: Cao Dong <87467313+dcaox@users.noreply.github.com>	2025-09-30 09:32:37 +08:00
Cheng Hang	cdce68c3e0	[TRTLLM-6741][fix] Add heuristics for lm head tp size when `enable_lm_head_tp_in_adp=True` (#7891 ) Signed-off-by: Cheng Hang <chang@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>	2025-09-30 09:24:35 +08:00
mpikulski	31a1a5ff80	[TRTLLM-8269][test] do not explicitly pass temperature=0 to select greedy sampling (#7909 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-09-29 14:52:18 +01:00
bhsueh_NV	38d6e4e60b	[None][feat] Support Qwen3 next (#7892 ) Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com> Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Co-authored-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-09-29 21:16:07 +08:00
mpikulski	a0d489a8d5	[TRTLLM-7728][perf] improve batched sampling perf for contiguous batches (#7908 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-09-29 13:32:50 +01:00
Yiqing Yan	560ded5450	[None][chore] Bump version to 1.2.0rc0 (#7941 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-09-29 17:39:07 +08:00
Gal Hubara-Agam	b2095aa074	[#4674 ][bugfix] AutoDeploy Fix memory leak in fuse_moe (#7844 ) Delete the unstacked weights immediately to save GPU memory, cleanup occurs automatically after the transformation, but for large models we'll run out of memory during the transformation itself. Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com>	2025-09-29 11:01:07 +03:00
Void	7f1e2dba92	[None][fix] only support deepep post quant all2all on nvfp4 (#8041 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-09-29 14:37:50 +08:00
Tailing Yuan	985b79ca82	[TRTLLM-8348][feat] Speed up concat k and copy k_nope in context phase using torch.compile (#8044 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-09-29 13:28:12 +08:00
Eran Geva	9cea6bfb30	[#7288 ][feat] Added AutoDeploy backend support to test_perf.py (#7588 ) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>	2025-09-28 21:21:27 -07:00
Zongfei Jing	e9f26feeb6	[None][chore] Cherry-pick from (#7598 ) Make low_precision_combine as a llm arg (#7898 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-09-28 22:32:33 -04:00
Yukun He	28b9a81c58	[TRTLLM-4500][feat] Add serialization/deserialization options for AutoTuner profiling cache (#7738 ) To achieve determinism for the AutoTuner profiling cache, serialization and deserialization are introduced to store the cache on disk in JSON format. Use TLLM_AUTOTUNER_CACHE_PATH to indicate the path where the cache file should be stored: Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-09-29 07:40:51 +08:00
Guoming Zhang	3ba4bf6e70	[None][chore] Disable concurrent weights loading for _load_weights_im… (#8034 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-09-28 07:11:16 -04:00
ChristinaZ	95eac2cda7	[https://nvbugs/5537738 ][fix] Add fp8 post-quant allgather support (#8008 ) Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>	2025-09-28 15:32:45 +08:00
Aurelien Chartier	77b68d9d7d	[https://nvbugs/5461712 ] [fix] Use DG for Qwen3 Linear layers (#8030 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-09-28 10:33:36 +08:00
Xianjie Qiao	c8f98b3065	[None] [feat] Update disagg gen-only benchmark. (#7917 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-09-28 09:56:56 +08:00
Iman Tabrizian	33282351a2	[TRTLLM-6106][feat] Add support for KVCache transfer from KVCache reuse path (#6348 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-09-27 19:29:30 -04:00
Frida Hou	a36b48bcab	[#5860 ][autodeploy] GPT-OSS MXFP4 support (#7451 ) Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>	2025-09-26 15:36:06 -07:00
Jhao-Ting Chen	c33f43e13a	[https://nvbugs/5518713 ][fix] Trtllm-gen moe backend for blockwise fp8 ckpt (Qwen3-235B-A22B-FP8) (#7856 ) Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>	2025-09-26 14:29:32 -07:00
Mike Iovine	d7087015f1	[TRTLLM-8271][fix] Fix CDL overlap scheduling performance (#7971 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-09-26 16:05:10 -04:00
YueWeng	a4243f0da5	[TRTLLM-6393][feat] add static tree sampling and verification (#7161 ) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>	2025-09-26 13:16:16 -04:00
HuiGao-NV	f4d3be4bbc	[None][feat] Add a standalone buffer cache class and reuse buffers between cduagraph and no-graph flow (#7669 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-09-26 07:28:06 -07:00
Tailing Yuan	b11ee868c5	[https://nvbugs/5495789 ][feat] Optionally disable server GC and worker GC (#7995 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-09-26 21:39:24 +08:00
HuiGao-NV	a9965d84e0	[None][chore] Report NCCL error message but not OOM when NCCL error happens (#8009 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-09-25 23:07:32 -07:00
peaceh-nv	55ce70060e	[https://nvbugs/5451740 ][fix] Add DP padding back on SM120 (#7965 ) Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>	2025-09-26 13:59:54 +08:00
Lucas Liebenwein	3a96d75a3c	[https://nvbugs/5527956 ][fix] AutoDeploy: fix IMA due to outdated metadata (#8002 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-09-25 22:05:55 -07:00
sunnyqgg	2e5850c28a	[TRTLLM-7330][feat] Eagle3 cuda graph support for the first draft model inference (#7363 ) Signed-off-by: qgai <qgai@nvidia.com>	2025-09-26 11:28:05 +08:00
Yuan Tong	fae83c387b	[#6102 ][fix] support non-system python installation (#7763 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-09-26 10:16:15 +08:00
Yanchao Lu	7e2521a7f0	[None][chore] Some clean-ups for CUDA 13.0 dependencies (#7979 ) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>	2025-09-26 08:46:11 +08:00
dongfengy	1eb653146a	[https://nvbugs/5525951 ][fix] Clarify that PP is not supported for GPTOSS (#7911 ) Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>	2025-09-25 12:54:18 -07:00
QI JUN	1529a6f22d	[None][chore] extract weights loading related logic to model loader (#7579 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-09-25 10:19:22 -07:00
xxi	57ff5f4c0d	[None][fix] fix a bug in wideEp use DeepEP with num_chunks > 1 (#7954 ) Signed-off-by: xxi <xxi@nvidia.com>	2025-09-25 07:53:42 -07:00
Matthias Jouanneaux	eda1467061	[TRTLLM-5966][feat] Helix: add alltoall op (#6815 ) Signed-off-by: Matthias Jouanneaux <mjoux@nvidia.com>	2025-09-25 07:18:29 -07:00
Yueh-Ting (eop) Chen	c5012423f5	[None][chore] Remove developer name in comment (#7981 ) Signed-off-by: eopXD <yuehtingc@nvidia.com>	2025-09-25 06:43:38 -07:00
Guoming Zhang	202bed4574	[None][chroe] Rename TensorRT-LLM to TensorRT LLM for source code. (#7851 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
QI JUN	961418908c	[https://nvbugs/5531963 ][fix] cherry pick #7725 (#7907 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
Yan Chunwei	cb466a846d	[None][fix] api stability bug in status label (#7861 ) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
Yan Chunwei	9d48898def	[None][doc] add stable label to all the un-labelled arguments in LLM class (#7863 ) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
Guoming Zhang	9f0f52249e	[None][doc] Rename TensorRT-LLM to TensorRT LLM for homepage and the … (#7850 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
Yan Chunwei	5342c607cd	[https://nvbugs/5516710 ][fix] fix Llama 3.3 TP PP case (#7717 ) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
Tao Li @ NVIDIA	44d7c3b245	[https://nvbugs/1234567 ][fix] Revert https://github.com/NVIDIA/TensorRT-LLM/pull/7768/files (#7813 ) Signed-off-by: Tao Li Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-25 21:02:35 +08:00
Wanli Jiang	22b45ff9c7	[TRTLLM-7758][feat] Phi4-mm image modality inference optimization (#7918 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-09-25 15:58:29 +08:00
Void	336c2ef540	[None][feat] DeepEP LL fp8 dispatch/combine (#7927 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-09-25 09:20:24 +08:00
Leslie Fang	342014069e	[None][chore] Validate features combination (#7630 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-25 08:01:13 +08:00
Iman Tabrizian	da30d496b0	[None][fix] Revert "[None][feat] Return topk logprobs in torch backend (#7756 )" (#7969 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-09-24 15:36:38 -07:00
sychen52	5a65af24cd	[OMNIML-2336][feat] Add NVFP4 x FP8 moe kernels (#7821 ) Signed-off-by: Shiyang Chen <shiychen@nvidia.com>	2025-09-24 12:14:35 -07:00
Mike Iovine	42c2ec3239	[https://nvbugs/5473781 ][fix] Fix llama 4 FP8 for PP>1 (#7220 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-09-24 12:16:27 -04:00
Yuxian Qiu	48fda86c56	[None][fix] Fix dummy load format for DeepSeek. (#7874 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-09-24 23:03:16 +08:00
Macrocell	6e5e8b8a3b	[None][fix] fix get_iteration_stats IndexError (#7216 ) Signed-off-by: yuhongwei <yumiao.yhw@antgroup.com> Co-authored-by: yuhongwei <yumiao.yhw@antgroup.com>	2025-09-24 22:43:03 +08:00
Eran Geva	603517f72a	[#7675 ][feat] CapturedGraph to support max_batch_size > max(cuda_graph_batch_sizes) (#7888 ) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>	2025-09-24 10:11:44 -04:00
Necofish	cfbcf9b9e8	[None][feat] Support Seed-OSS model in pytorch backend (#7496 ) Signed-off-by: Nekofish-L <liuxiangyang@mail.ustc.edu.cn>	2025-09-24 03:57:12 -07:00
Enwei Zhu	a1a57e83b8	[TRTLLM-5235][feat] Enable regex and EBNF grammar in trtllm-serve (#7925 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-24 18:30:23 +08:00
JunyiXu-nv	6654b78c94	[https://nvbugs/5521799 ][fix] Trim incorrectly generated harmony messages (#7849 ) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>	2025-09-24 16:38:43 +08:00
Cao Dong	2f8dc6feb0	[None][feat] Return topk logprobs in torch backend (#7756 ) Signed-off-by: Dong Cao <docao@nvidia.com>	2025-09-24 15:30:39 +08:00
Yueh-Ting (eop) Chen	cf100933cc	[TRTLLM-6341][feature] Support SWA KV cache reuse (#6768 ) This merge request attempts to support more SWA KV cache functionality inside the KV cache manager. Before this merge request, the KV cache for sliding window attention (SWA) only holds "window size" number of blocks and reuse them in a cyclic manner. We will not be able to utilize more GPU memory with this design, leading to a limited max batch size throughput. Additionally, we will not be able to support KV cache reuse with this design. In this MR, we change such behavior to let the manager write blocks in a linear manner. With a linear block writing behavior, as the attention window moves on, the out-of-window (OOW) blocks will be detached. Right now for the sake of a correct feature first, we directly offload the OOW block from the primary block pool (GPU memory) to the secondary block pool (host memory). We will improve this in the future by delegating the block movement to the eviction policy. KV cache reuse for SWA is not developed in this merge request and will be amended in a follow-up merge request. Writing the blocks linearly, the maximum number of blocks allocated for a sequence(`GenerationRequest`) is the "max sequence length" specified. The `GenerationRequest` that stores the cache block bookkeeping structure will now keep "max sequence length" tokens of blocks. Given the above, main changes are (more context in the MR): - Remove "cyclic" concept under the kv cache manager, such concept originally guards the block reuse under kv cache manager. - Add detach mechanism and have it under `KVCacheManager::addToken`. Please note that detach is still guarded off for SWA when reuse is enabled. A follow-up merge request will proceed to improve this. - Enforce "max sequence length" to be a non-optional parameter to the `KVCacheManager`/`BlockManager` - Let all window size resource pool get identical proportion of memory - Fix free memory calculation under `resource_manager.py` Signed-off-by: eopXD <yuehtingc@nvidia.com> Co-authored-by: Tomer Asida <tasida@nvidia.com>	2025-09-24 14:28:24 +08:00
Daniel Cámpora	5ccb2dea33	[None][chore] Make sampler type beta. (#7934 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-09-23 20:51:39 -07:00
Yuan Tong	70c3b100eb	[#7692 ][fix] recognize RequestError as per-request error in background handler (#7726 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-09-24 11:11:17 +08:00
Yuan Tong	f050b8d871	[None][fix] refine `backend` option handling for commands (#7829 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-09-24 10:54:33 +08:00
Ziyi Xiong	31ef03fd82	[https://nvbugs/5528405 ][fix] Set up draft_tokens before scheduling (#7903 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-09-24 09:56:17 +08:00
Venky	6ff0fad75e	[TRTLLM-7015] [feat] Enable `prompt_logprobs` in pytorch backend (#7580 ) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>	2025-09-23 18:48:10 -07:00
Lizhi Zhou	7550251988	[TRTLLM-7182][test] add multi-nodes test for disagg-serving (#7470 ) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>	2025-09-24 08:31:56 +08:00
mpikulski	9970345919	[TRTLLM-7728][feat] batched sampling by strategy (supersedes enable_mixed_sampler, cf. TRTLLM-7156) (#7294 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-09-23 16:05:05 -07:00
Yilin Fan	7d4d6cc9e0	[TRTLLM-7292][feat] Support multi-threaded tokenizers for trtllm-serve (cherry-pick) (#7776 ) Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>	2025-09-23 09:39:47 -07:00
Daniel Cámpora	9f1d9b7b18	[None][feat] Use list instead of torch tensor for new tokens in update requests (#7730 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-09-23 10:40:08 -04:00
Zheyu Fu	34963ec39c	[None][fix] Assign [] to req.py_draft_tokens instead of None when spec decode is off (#7511 ) Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>	2025-09-23 06:54:18 -07:00
ChristinaZ	dd5fb2857a	[None][fix] Re-add the import for allgather that was mistakenly removed. (#7920 ) Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>	2025-09-23 03:09:48 -07:00
Yan Chunwei	3ba19b6ff1	[https://nvbugs/5532023 ][fix] executor with-statement bug (#7895 ) Signed-off-by: chunweiy <chunweiy@nvidia.com>	2025-09-23 02:05:39 -07:00
Enwei Zhu	f882fb86db	[https://nvbugs/5367180 ][fix] Fix xgrammar import before loading tensorrt_llm binary (#7906 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-23 00:29:57 -07:00
Yan Chunwei	40820e6711	[None][fix] CHERRY-PICK trtllm-serve yaml loading (#7551 ) (#7897 ) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>	2025-09-23 14:56:52 +08:00
Pengbo Wang	5792464d37	[None][fix] Read eos_token_id from generation_config for kimi_k2 (#7120 ) Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>	2025-09-23 10:47:03 +08:00
yunruis	126cd707e3	[None][opt] Add batch waiting when scheduling (#7416 ) Signed-off-by: yunruis <205571022+yunruis@users.noreply.github.com> Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com>	2025-09-23 10:27:37 +08:00
Chang Liu	998857bcde	[TRTLLM-7328][feat] E-PD Disagg Support via llmapi (3/N) (#7577 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-09-22 19:07:18 -07:00
jianweiwu	9da4203e2e	[None][feat] Add Tencent HunYuanDenseV1 model support (#7081 ) Signed-off-by: sorenwu <sorenwu@tencent.com> Signed-off-by: jianweiwu <sorenwu@tencent.com> Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-23 09:27:29 +08:00
Tailing Yuan	740340dd17	[https://nvbugs/5522847 ][fix] Disable GC on disagg server and client (#7858 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-09-23 09:16:55 +08:00
Enwei Zhu	8330d5363a	[TRTLLM-8209][feat] Support new structural tag API (upgrade XGrammar to 0.1.25) (#7893 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-23 09:10:09 +08:00
xxi	d471655242	[TRTLLM-7831][feat] Cherry-pick from #7423 Support fp8 block wide ep cherry pick (#7712 )	2025-09-23 08:41:38 +08:00
Enwei Zhu	59f57598a7	[https://nvbugs/5504086 ][fix] Fix MTP vanilla (#7904 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-23 08:38:28 +08:00
ChristinaZ	be576a3152	[None] [feat] Enable run_post_quant_allgather for MoE TRTLLM backend (#6794 ) Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>	2025-09-23 08:24:21 +08:00
Jin Li	b5391b4ac6	[https://nvbugs/5516665 ][fix] Fix CUTLASS moe fake impl errors (#7714 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-09-22 11:08:39 -07:00
Wanli Jiang	2a30f11d63	[None][chore] Upgrade transformers to 4.56.0 (#7523 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com> Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-22 22:20:16 +08:00
Yechan Kim	f77aca9f2c	[TRTLLM-7385][feat] Optimize Qwen2/2.5-VL performance (#7250 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-09-22 03:40:02 -07:00
HuiGao-NV	0dac1ddb74	[https://nvbugs/5525849 ][fix] Cherry-pick to fix mismatch of max seq len between kv cache manager and dummy requests (#7855 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-09-22 18:07:47 +08:00
Yukun He	ab26d21620	[https://nvbugs/5517023 ][fix] Pass allreduce strategy and force NCCL on pre-Blackwell arch (#7768 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
Yan Chunwei	ba2864a2c6	[None][doc] Enhance api reference doc by labeling stable APIs (#7751 ) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
Yi Zhang	f9c9c3f50a	[https://nvbugs/5355219 ][fix] Fix trtllm moe backend test config and Qwen3 MoE multi node (#7724 ) Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
HuiGao-NV	af34c9713a	[https://nvbugs/5474169 ][fix] seq_len mismatch between kv cache manager and graph attn metadata (#7606 ) Signed-off-by: Hui Gao <huig@nvidia.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
Yukun He	3cc16c2438	[https://nvbugs/5496960 ][fix] Fix Gemma model forward. (#7509 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
Yuxian Qiu	2d46dda6a7	[https://nvbugs/5448754 ][fix] Download HF model for all nodes. (#6824 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
HuiGao-NV	123f5cbbf0	[https://nvbugs/5474169 ][fix]Adjust max seq len for kvcache for memory estimation (#7391 ) Signed-off-by: Hui Gao <huig@nvidia.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
Bo Li	a15f08db3d	[https://nvbugs/5467548 ][fix] DeepSeek illegal memory access. (#7298 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-22 14:28:38 +08:00
Stefan Niebler	8aead224fb	[https://nvbugs/5513423 ][fix] Correctly respect min_tokens in PyTorch Workflow (#7808 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com> Co-authored-by: Daniel Cámpora <961215+dcampora@users.noreply.github.com>	2025-09-21 22:15:18 -07:00
dongxuy04	b057fc9593	[None][fix] cherrypick to main: Fix possible mpi broadcast and gather issue on large object (#7854 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-09-22 10:17:23 +08:00
Enwei Zhu	639d4109a7	[None][fix] Disable torch.compile for CapturableGuidedDecoder (#7871 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-22 10:04:30 +08:00
dongxuy04	9eb8084ca9	[TRTLLM-7008][fix] cherrypick to main Add automatic shared memory delete if already exist (#7727 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-09-21 11:01:51 -07:00
Ziyi Xiong	897c4dd23b	[https://nvbugs/5517404 ][fix] Use the correct cuda graph for dynamic spec dec (#7728 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-09-21 08:20:48 +08:00
Yan Chunwei	4509d97780	[TRTLLM-8188][chore] refactor GenerationExecutorWorker with WorkerBase for better code reusing (#7840 ) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>	2025-09-20 06:24:22 -07:00
Grzegorz Kwasniewski	8adaf0bb78	[TRTLLM-6342][feat] Support for partial sharding from factory (#7393 ) Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> Signed-off-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com>	2025-09-19 09:07:42 -07:00
Matthias Jouanneaux	1be7faef37	[TRTLLM-5966][feat] Helix: add custom position ids to MLA kernels (#6904 ) Signed-off-by: Matthias Jouanneaux <mjoux@nvidia.com> Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com>	2025-09-19 20:55:32 +08:00
Liao Lanyu	18095a7cb8	[https://nvbugs/5503440 ][fix] Fix potential hang due to wrong type of ZMQ socket and protocol for worker_init_status_queue (#7646 ) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>	2025-09-19 18:13:33 +08:00
Gabriel Wu	0e72e8f7e6	[None][feat] Support EPLB in Qwen3 MoE (#7443 ) Signed-off-by: Gabriel Wu <13583761+lucifer1004@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2025-09-19 16:45:35 +08:00
QI JUN	f1b362faac	[None][chore] polish error message in cute_dsl_utils.py (#7852 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-09-19 12:05:11 +08:00
HuiGao-NV	a6370fd143	[https://nvbugs/5481434 ][feat] cherry-pick fix to reuse pytorch memory segments occupied by cudagraph (#7747 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-09-19 10:25:21 +08:00
Yuxian Qiu	d6ebcf7c4a	[TRTLLM-6994][feat] FP8 Context MLA integration (Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6059 from release/1.1.0rc2) (#7610 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-09-19 09:40:49 +08:00
Ziyi Xiong	420f0fbcf5	[https://nvbugs/5522851 ][fix] Correct the logic to update kv_lens_cuda (#7790 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-09-19 08:11:29 +08:00
sunnyqgg	80dd8fe197	[TRTLLM-6746][feat] Enable two-model spec dec for MTP Eagle (#7001 ) Signed-off-by: qgai <qgai@nvidia.com>	2025-09-18 12:05:36 -04:00
Li Min	d921fc3352	[TRTLLM-6898][feat] Add swapab, tileN64, cga sync support for cute dsl nvfp4 gemm (#7764 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-09-18 21:20:04 +08:00
bhsueh_NV	c65457db8a	[None][fix] Revert "Revert "[None][feat] support attention dp for qwen3 dense model"" (#7780 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-09-18 20:11:05 +08:00
Wanli Jiang	fe104dc20d	[TRTLLM-7918][feat] Support kvcache reuse and chunk prefill for phi4mm (#7723 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-09-18 17:37:16 +08:00
Stefan Niebler	a55251bf75	[None][fix] Add TP information in weight scale loading in WeightOnlyQuantLinearMethod (#7732 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-09-18 10:30:50 +02:00
Wanli Jiang	a7ca0fff54	[TRTLLM-6577][feat] Support nano_v2_vlm in pytorch backend (#7207 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-09-18 16:26:20 +08:00
Leslie Fang	870cfcf9a0	[None][chore] Remove executor config in create_py_executor (#7599 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-18 14:24:58 +08:00
mpikulski	1c7f601265	[https://nvbugs/5508890 ][fix] gen. result cleanup when using PostprocWorker (#7771 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-09-18 14:01:18 +08:00
Li Min	14e455da3e	[None][fix] Fix CI issue for dsl pkg install (#7784 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com> Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-09-18 13:58:20 +08:00
Barry Kang	4f0e6b5f96	[None][feat] Cherry-pick DeepGEMM related commits from release/1.1.0rc2 (#7716 ) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>	2025-09-18 13:51:48 +08:00
Ziyi Xiong	28469dbf27	[https://nvbugs/5523080 ][fix] Correct the batch index in device tensors (#7803 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-09-18 13:45:37 +08:00
Guoming Zhang	e0423bfaab	[https://nvbugs/5519544 ][fix] fix invalid expression for disabling pa… (#7806 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-09-18 12:54:52 +08:00
Yanchao Lu	f8e811d134	[None][chore] Version bump for 1.1.0rc6 (#7824 ) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>	2025-09-18 11:13:56 +08:00
Yukun He	cd80e0a7f1	[None][fix] Make tile_tokens_dim calculation just in time before kernel launching. (#7529 ) tile_tokens_dim directly depends on the num_token, which is a dynamic shape during tuning and inference. When AutoTuner prepares dummy tensors with different num_tokens, it does not update the value of tile_tokens_dim automatically. Therefore, the value stored in the AutoTuner cache is misaligned, which will introduce a lot of cache misses during inference, which hurts perf a lot. To avoid this issue, we move the calculation of tile_tokens_dim right before kernel launching, so that the value of tile_tokens_dim is always up to date with the num_tokens of the current input tensor used for the kernel runner. Also, the tile_tokens_dim is calculated based on the number of tokens of a tuned bucket, instead of the original token number. Because we only tune the value for the buckets, not for the raw input token number, to avoid unexpected misalignment between tile_tokens_dim and the token number. This PR also removes the warmup requests with the extra input shapes, which are triggered in the CUDA graph warmup phase. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-09-18 10:58:52 +08:00
Lucas Liebenwein	39eb120b96	[#7308 ] [feat] AutoDeploy: graph-less transformers mode for HF (#7635 ) Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>	2025-09-18 10:44:24 +08:00
Netanel Haber	a5cfc8368f	[https://nvbugs/5508536 ][fix] Revert #7041 : Move stop_criteria to sample_async (#7041 ) (#7796 ) Signed-off-by: Netanel Haber <nhaber@nvidia.com> Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com> Co-authored-by: Mike Iovine <miovine@nvidia.com>	2025-09-17 21:27:01 -04:00
William Zhang	2614d71994	[TRTLLM-7410][feat] Enable KV cache reuse and chunked prefill for mistral3.1 (#7628 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-09-17 08:11:16 -07:00
Zhenhuan Chen	6983e8a00d	[https://nvbugs/5517260 ][fix] move scaffolding contrib module's import to subdirectory (#7758 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-09-17 11:36:33 +08:00
Kaiyu Xie	62042a9733	[TRTLLM-6741] [feat] enable LM tp for MTP, under attention dp case (cherry-pick #7128 ) (#7571 ) Signed-off-by: Cheng Hang <chang@nvidia.com> Co-authored-by: Cheng Hang <chang@nvidia.com>	2025-09-17 09:41:32 +08:00
Yukun He	6313c9799c	[https://nvbugs/5488582 ][fix] Cherry-pick 7495: Avoid unexpected Triton recompilation in DG fused_moe (#7708 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-09-17 09:00:28 +08:00
Shiyu Li	8bdbb48264	[https://nvbugs/5489015 ][fix] Support communicator split in MNNVL allreduce and fix the binding issues. (#7387 ) Signed-off-by: Shiyu Li <shili@nvidia.com>	2025-09-17 07:43:20 +08:00
HuiGao-NV	a49cfb3e68	[https://nvbugs/5516666 ][fix] cherrypick fix to the CUDA graph warmup issue when using speculative decoding (#7737 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com> Co-authored-by: Signed-off-by: Hui Gao <huig@nvidia.com>	2025-09-17 06:24:20 +08:00
Aurelien Chartier	471723bce1	[None][chore] Remove unused get_quant_scales methods (#7687 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-09-16 12:56:11 -07:00
Lucas Liebenwein	9befd1a72f	[None][chore] AutoDeploy: neat disablement of transforms in pipeline (#7736 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-09-16 23:31:48 +08:00
bhsueh_NV	8226ef23dc	Revert "[None][feat] support attention dp for qwen3 dense model" (#7765 )	2025-09-16 19:09:04 +08:00
Kaiyu Xie	6eef19297f	[None] [chore] cherry pick changes on slurm scripts from `release/1.1.0rc2` (#7750 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-09-16 16:07:13 +08:00
Li Min	b278d06481	[TRTLLM-6898][feat] Add Cute DSL nvfp4 linear op (#7632 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-09-16 14:25:26 +08:00
Bo Li	3f4e160cba	[None][chore] Fix error when running trtllm-bench without cuda graph. (#7725 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-09-15 20:30:23 -07:00
Void	103b554734	[None][fix] Ensure that the W4A8 custom input scale remains aligned across all ranks (#7614 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-09-16 11:04:26 +08:00
Yanchao Lu	e5cead1eb9	[TRTLLM-6295][test] Exit as early as possible and propagate exit status correctly for multi-node testing (#7739 ) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>	2025-09-16 09:59:18 +08:00
xiweny	c076a02b38	[TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568 ) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com> Signed-off-by: Tian Zheng <29906817+Tom-Zheng@users.noreply.github.com> Signed-off-by: Daniel Stokes <dastokes@nvidia.com> Signed-off-by: Zhanrui Sun <zhanruis@nvidia.com> Signed-off-by: Xiwen Yu <xiweny@nvidia.com> Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com> Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> Signed-off-by: Bo Deng <deemod@nvidia.com> Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Signed-off-by: xiweny <13230610+VALLIS-NERIA@users.noreply.github.com> Co-authored-by: Tian Zheng <29906817+Tom-Zheng@users.noreply.github.com> Co-authored-by: Daniel Stokes <dastokes@nvidia.com> Co-authored-by: Zhanrui Sun <zhanruis@nvidia.com> Co-authored-by: Jiagan Cheng <jiaganc@nvidia.com> Co-authored-by: Yiqing Yan <yiqingy@nvidia.com> Co-authored-by: Bo Deng <deemod@nvidia.com> Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-09-16 09:56:18 +08:00
Necofish	96f11b10ae	[None][feat] support attention dp for qwen3 dense model (#7618 ) Signed-off-by: Nekofish-L <liuxiangyang@mail.ustc.edu.cn>	2025-09-16 09:33:22 +08:00
Ziyi Xiong	536e8776cd	[TRTLLM-6668][feat] Enable overlap scheduler for two-model spec decoding (#7651 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-09-16 07:33:44 +08:00
Izzy Putterman	8097be7e9c	[None][feat] Eagle, use last hidden post norm (#7546 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-09-15 12:23:57 -04:00
jmydurant	7deefb3d2b	[TRTLLM-7192][feat] optimize MLA chunked prefill && support fp8 mla chunked prefill (#7477 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-09-15 21:43:49 +08:00
Zheng Duan	24fc1f9acf	[None][fix] using arrival time in llmapi when creating LlmRequest in pytorch workflow (#7553 ) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>	2025-09-15 07:26:01 -04:00
Wanli Jiang	e080294725	[TRTLLM-7918][feat] Revert "Support kvcache reuse for phi4mm (#7563 )" (#7722 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-09-15 17:19:44 +08:00
Wanli Jiang	fc9f4c9295	[TRTLLM-7918][feat] Support kvcache reuse for phi4mm (#7563 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-09-15 15:47:00 +08:00
DylanChen-NV	d5df0af017	[https://nvbugs/5467981 ][fix] Fix Qwen2.5-VL fails with cuda graph padding (#7122 ) Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>	2025-09-15 15:02:34 +08:00
Chang Liu	47e37755a3	[TRTLLM-6903][feat] Support chunked prefill for multimodal models (#6843 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-09-14 20:10:10 -07:00
Pengyun Lin	c2bc39af63	[TRTLLM-1302][feat] Topk logprobs for TRT backend and top1 logprob for PyT backend (#6097 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-09-12 15:32:34 +08:00
Chang Liu	3a9847eb84	[https://nvbugs/5498165 ][fix] fix permission error for config file lock (#7656 ) Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>	2025-09-11 10:36:51 +08:00
Dom Brown	fc9d426589	[https://nvbugs/5505402 ] [fix] Disable deep_gemm for Qwen3 QKNormRoPEAttention and Linear layers due to accuracy issues (#7616 ) Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-09-10 18:30:48 +01:00
Leslie Fang	d219a4f225	[None][chore] remove executor config in kv cache creator (#7526 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-10 21:14:44 +08:00
Yiqing Yan	76c5e1a12f	[None][infra] Bump version to 1.1.0rc5 (#7668 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-09-10 16:06:54 +08:00
Kanghwan	758c22f832	[#7208 ][fix] Fix config type of MedusaConfig (#7320 ) Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>	2025-09-09 23:25:17 -07:00
Frida Hou	bbb5ae3349	[#5861 ][autodeploy] Refactor: Quantization Transforms with Inheritance (#7227 ) Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-09-10 13:00:06 +08:00
Zheyu Fu	c353ff342e	[None][feat] Make the should_use_spec_decode logic a bit smarter (#7112 ) Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>	2025-09-10 12:53:59 +08:00
Chang Liu	faa2f46554	[TRTLLM-5059][feat] Enable KV-cache reuse and add E2E tests for llava-next (#7349 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-09-09 14:51:36 -04:00
Jin Li	d49374bc45	[TRTLLM-7408][feat] Wrap MOE with custom op. (#7277 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-09-09 12:18:56 -04:00
Richard Huo	dcd110cfac	[None][chore] add TorchLlmArgs to the connector api (#7493 ) Signed-off-by: richardhuo-nv <rihuo@nvidia.com>	2025-09-09 09:05:59 -04:00
NVJiangShao	cc7593987b	[https://nvbugs/5434424 ][fix] A quick fix for the wrong output issue of SM89 blocked scaling batched GEMM when the input tensor is non-contiguous. (#7615 ) Signed-off-by: Jiang Shao <91270701+StudyingShao@users.noreply.github.com>	2025-09-09 08:58:15 -04:00
tomeras91	6e712dd1cc	[None][fix] enable NvFP4/FP8 quantization for Nemotron-H architecture (#7589 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-09-09 11:42:22 +03:00
Linda	9cb5410067	[https://nvbugs/5454559 ][fix] handle bias term in fuse_gate_mlp (#7449 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-09-09 10:26:17 +02:00
William Zhang	c53d1814a7	[None][feat] Extend VLM factory and add Mistral3 factory (#7583 ) This commit: * extends existing factory interfaces to enable Mistral3 in AutoDeploy. * adds a Mistral3 VLM factory. * adds various model patches for pixtral (the vision model) and mistral3 to make the VLM export compliant. * adjusts checkpoint loading code to take possible parameter name conversions into account. * fixes a sampling bug (the `end_id` needs to be take into account when sampling, but it is not included in the stop words' token IDs). Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-09-09 02:47:18 -04:00
Guoming Zhang	f53fb4c803	[TRTLLM-5930][doc] 1.0 Documentation. (#6696 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-09 12:16:03 +08:00
zhanghaotong	96af324ff1	[None][fix] Add try-catch in stream generator (#7467 ) Signed-off-by: Zhang Haotong <zhanghaotong.zht@antgroup.com> Co-authored-by: Zhang Haotong <zhanghaotong.zht@antgroup.com>	2025-09-08 16:09:26 -04:00
Chuang Zhu	77657a1c12	[TRTLLM-7361][feat] KV cache transfer for uneven pp (#7117 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-09-08 13:37:46 -04:00
Leslie Fang	3e0073e86b	[None][chore] remove executor config in instantiate sampler (#7516 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-08 09:02:40 -07:00
Eran Geva	5f2a42b3df	[TRTLLM-6142][feat] AutoDeploy: set torch recompile_limit based on cuda_graph_batch_sizes and refactored (#7219 ) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>	2025-09-08 08:45:58 -04:00
Chang Liu	4a1e13897f	[None][feat] Update multimodal utility `get_num_tokens_per_image` for better generalization (#7544 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-09-08 07:42:46 -04:00
dominicshanshan	c9dca69e1b	[None][chore] Mass integration of release/1.0 - 3rd (#7519 ) Signed-off-by: Nave Assaf <nassaf@nvidia.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com> Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> Signed-off-by: qqiao <qqiao@nvidia.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Bo Deng <deemod@nvidia.com> Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com> Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com> Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com> Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> Signed-off-by: Pamela <179191831+pamelap-nvidia@users.noreply.github.com> Signed-off-by: Hui Gao <huig@nvidia.com> Signed-off-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com> Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Michal Guzek <mguzek@nvidia.com> Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com> Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com> Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com> Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com> Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com> Co-authored-by: Nave Assaf <55059536+Naveassaf@users.noreply.github.com> Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com> Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Co-authored-by: Emma Qiao <qqiao@nvidia.com> Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Co-authored-by: Bo Deng <deemod@nvidia.com> Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com> Co-authored-by: yifeizhang-c <219273404+yifeizhang-c@users.noreply.github.com> Co-authored-by: amitz-nv <203509407+amitz-nv@users.noreply.github.com> Co-authored-by: Erin <14718778+hchings@users.noreply.github.com> Co-authored-by: chenfeiz0326 <chenfeiz@nvidia.com> Co-authored-by: ChristinaZ <83400082+ChristinaZ@users.noreply.github.com> Co-authored-by: Venky <23023424+venkywonka@users.noreply.github.com> Co-authored-by: Pamela Peng <179191831+pamelap-nvidia@users.noreply.github.com> Co-authored-by: HuiGao-NV <huig@nvidia.com> Co-authored-by: milesial <milesial@users.noreply.github.com> Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: Michal Guzek <moraxu@users.noreply.github.com> Co-authored-by: peaceh-nv <103117813+peaceh-nv@users.noreply.github.com> Co-authored-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com> Co-authored-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com> Co-authored-by: pcastonguay <55748270+pcastonguay@users.noreply.github.com> Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com> Co-authored-by: Linda <57756729+Linda-Stadter@users.noreply.github.com> Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Co-authored-by: Jiagan Cheng <jiaganc@nvidia.com> Co-authored-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com> Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com> Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-09-08 14:03:04 +08:00
JunyiXu-nv	504bb7ffa9	[TRTLLM-7779][feat] Support multiple postprocess workers for chat completions API (#7508 ) Signed-off-by: Junyi Xu Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-09-08 11:11:35 +08:00
Yan Chunwei	205c3a144c	[None][chore] expose tokens_per_block into KvCacheConfig (#5911 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>	2025-09-07 21:14:10 -04:00
Netanel Haber	0fee8cd028	[TRTLLM-7153] [feat] Move stop_criteria to sample_async (#7041 ) Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-09-07 17:36:49 +03:00
Raayan Dhar	bae9560e62	[https://nvbugs/5448767 ][fix] sync termination of requests across PP ranks (#7455 ) Signed-off-by: raayandhar <rdhar@nvidia.com> Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> Co-authored-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>	2025-09-07 08:45:49 -04:00
Mike Iovine	45390402fc	[https://nvbugs/5502352 ][fix] Fix 2-model CDL path (#7543 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-09-06 23:53:27 -04:00
Chang Liu	99b98f1374	[TRTLLM-7440][fix] Split `fused_input_embed` to separate out host sync (#7280 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-09-06 23:11:39 -04:00
Chang Liu	23500b55c3	[TRTLLM-7398][feat] Support KV cache salting for secure KV cache reuse (#7106 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>	2025-09-06 17:58:32 -04:00
QI JUN	12ecb864c2	[None][chore] share input_ids buffers among different cuda graphs (#7236 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-09-06 17:49:42 -04:00
Anthony Chang	12c66f7610	[None][fix] DeepSeek-R1 W4A8 weight loading issue; fixes regression from #6200 (#7123 ) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>	2025-09-07 00:04:56 +08:00
Lucas Liebenwein	74105a45d9	[#6120 ][feat] AutoDeploy: flexible args for sequence interface + AD multi-modal input processor + llama4 VLM example (#7221 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-09-05 22:10:48 -04:00
Leslie Fang	9eb3911470	[None][chore] Remove executor_config in create_py_executor_instance (#7463 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-05 20:56:03 +08:00
Robin Kobus	a95d9616ba	[#6186 ][feat] Introduce QKNormRoPEAttention module (#6830 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-09-05 14:04:41 +02:00
Jin Li	2189a2f3ff	[https://nvbugs/5483615 ][fix] Remove unnecessary assertion to let mai… (#7441 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-09-05 10:56:21 +08:00
Naveenraj Kamalakannan	58d1036bb1	[#3325 ][feat] Add MCTS and TOT tree-based inference controllers to Scaffolding (#7490 ) Signed-off-by: Naveenraj Kamalakannan <therealnaveenkamal@gmail.com>	2025-09-04 19:46:49 -07:00
Shunkangz	bddf183e15	[None][feat] Add Request specific exception (#6931 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-09-04 18:43:42 -04:00
Rashid Kaleem	89889fb526	[https://nvbugs/5369366 ] [fix] Report failing requests (#7060 ) Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>	2025-09-04 12:56:23 -07:00
Chang Liu	08a0e06621	[TRTLLM-7410][feat] Support hashing and KV cache reuse for videos (#7360 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>	2025-09-04 14:39:23 -04:00
sychen52	98a1bffb7c	[OMNIML-2336][feat] Add NVFP4 x FP8 (#6809 ) Signed-off-by: Shiyang Chen <shiychen@nvidia.com>	2025-09-04 09:03:38 -07:00
Enwei Zhu	1745102e72	[TRTLLM-7027][feat] Fuse d2t to logitsBitmaskKernel and fix a race condition in one-model spec (#7481 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-09-04 23:30:14 +08:00
Izzy Putterman	26b133f3a7	[None][feat] MultiLayer Eagle (#7234 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-09-04 10:49:13 -04:00
Wanli Jiang	4e3dded64d	[TRTLLM-6308][feat] Support Aggregate mode for phi4-mm (#7521 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-09-04 20:16:10 +08:00
WeiHaocheng	5bcda7520b	[https://nvbugs/5477730 ][fix] Fix the alltoall case when tp_size larger than ep_size (#7331 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-09-04 08:10:03 -04:00
kris1025	cce9556858	[https://nvbugs/5485886 ][fix] Fix resource free of Eagle3ResourceManager (#7437 ) Signed-off-by: linquanh <linquanh@nvidia.com>	2025-09-04 17:38:13 +08:00
Yiqing Yan	ced5512ae4	[None][chore] Bump version to 1.1.0rc4 (#7525 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-09-04 16:30:47 +08:00
jianweiwu	7090b286b2	[None][fix] fix hunyuan_moe init bug (#7502 ) Signed-off-by: sorenwu <sorenwu@tencent.com>	2025-09-04 03:06:00 -04:00
Grzegorz Kwasniewski	3755f8ab7d	[TRTLLM-6342][fix] Fixed triggering BMM sharding (#7389 ) Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>	2025-09-04 02:01:27 -04:00
William Zhang	a117e7a57e	[TRTLLM-7442][model] Remove unnecessary D2H copies (#7273 ) * Why? Initial profiling showed there were multiple D2H / H2D copies being scheduled in the mistral 3.1 small model. * What? This commit removes those unnecessary copies by returning `image_sizes` as a simple list instead of a tensor. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-09-03 23:14:20 -04:00
Jin Li	2a2dfe273b	[https://nvbugs/5485102 ][fix] Correctly set stride for piecewise outp… (#7442 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-09-04 10:48:15 +08:00
Frida Hou	51a2b8729e	[#7222 ][autodeploy] Separate run_shape_prop as another graph utility (#7313 ) Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-09-03 19:32:50 -04:00
Leslie Fang	bd9ba97d89	[None][chore] Remove two unused parameters in create_py_executor (#7458 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-04 07:31:31 +08:00
Enwei Zhu	5ff3a65b23	[TRTLLM-7028][feat] Enable guided decoding with speculative decoding (part 2: one-model engine) (#6948 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-09-03 15:16:11 -07:00
Mike Iovine	64e3bfa054	[None][fix] Fix KV cache recompute in draft_target spec decode (#7348 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-09-03 15:04:14 -04:00
Anurag Mukkara	ae5136831f	[https://nvbugs/5472947 ][fix] wait on isend handles before reusing buffers (#7462 ) Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>	2025-09-03 13:20:02 +05:30
YueWeng	9a4f60687f	[https://nvbugs/5480289 ][fix] release slot manager in mtp MTPHiddenStatesManager (#7340 ) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>	2025-09-02 19:37:51 -07:00
Jinyang Yuan	572551b586	[None][perf] Autotune TRT-LLM Gen MoE when using CUDA graphs (#7285 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-09-03 10:08:59 +08:00
Leslie Fang	42697ea32a	[None][chore] rm executor config in kv cache connector (#7372 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-03 08:13:13 +08:00
JunyiXu-nv	eefe5f2093	[TRTLLM-7208][feat] Implement basic functionalities for Responses API (#7341 ) Signed-off-by: Junyi Xu <junyix@nvidia.com>	2025-09-02 07:08:22 -04:00
tomeras91	9c8d2161d0	[None][doc] fix example in docstring (#7410 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-09-02 11:59:49 +03:00
Leslie Fang	e81c50dbd2	[None][chore] Use llm args in create_py_executor (#7239 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-09-01 16:27:55 -07:00
Mike Iovine	b3c57a7042	[TRTLLM-7353][feat] Implement capturable drafting loops for speculation (#7100 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-09-01 14:37:44 -04:00
QI JUN	ed4087a295	[https://nvbugs/5374016 ][fix] improve error message (#6893 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
Aurelien Chartier	93e623b455	[https://nvbugs/5449155 ][fix] Fix DeepSeek R1 weight loading for TP16 (#6913 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
Liao Lanyu	704fca4178	[TRTLLM-6835][fix] Fix potential hang caused by python multiprocessing when prefetching weights (#6927 ) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
Mike Iovine	de55763f13	[https://nvbugs/5455836 ][fix] Fix llama 4 FP4 (#6911 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
brb-nv	0253036a4e	[None][chore] Add docs for Gemma3 VLMs (#6880 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
Yukun He	e106045fda	[None][fix] Complete the last missing allreduce op in Llama3/4. (#6850 ) The allreduce op of the last decoder layer is missing in some circumstances for the models Llama3 and Llama4. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
Anurag Mukkara	b821883b25	[None][fix] Revert phi4-mm aggregate mode (#6907 ) Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
2ez4bz	cf0c47ca2d	[None][fix] Fix batching bug in Mistral3 model (#6841 ) Prior to this commit, if multiple requests with images were in the same batch, the batching logic for the images would fail. This commit fixes it, and adds unit tests for it that were verified to fail prior to the fix. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
2ez4bz	2480aedb73	[TRTLLM-5252][feat] Add fp8 support for Mistral Small 3.1 (#6731 ) This commit adds some level of FP8 support to Mistral Small 3.1 by: * disabling quantization for the vision sub-model since `modelopt` does support quantizing it (yet). * extending existing accuracy tests to use a modelopt produced FP8 checkpoint. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-09-01 11:02:31 +08:00
Tian Zheng	e257cb3533	[None][feat] Support NVFP4 KV Cache (#6244 ) Signed-off-by: Tian Zheng <29906817+Tom-Zheng@users.noreply.github.com>	2025-09-01 09:24:52 +08:00
Zongfei Jing	a7ed26dd8b	[TRTLLM-6747][feat] Merge add sparse exp and shared exp into local reduction (#7369 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-08-31 21:20:00 -04:00
Yiqing Yan	ec595a8e29	[None][chore] Bump version to 1.1.0rc2 (#7394 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-31 10:20:38 +08:00
Zhongdongming Dai	9bb0c9500e	[None][docs] Update Dynasor paper info (#7137 ) Signed-off-by: Zhongdongming Dai <zhongdongmin@nvidia.com>	2025-08-29 18:47:47 -07:00
Fanrong Li	37a1bd810f	[https://nvbugs/5481385 ][fix] Fix max_seq_len in cuda graph warmup and intermediate_size in fused_moe_deepgemm (#7345 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com>	2025-08-29 17:00:43 +08:00
Chang Liu	31b0f0fb0c	[https://nvbugs/5445466 ][fix] Eliminate race when loading HF dynamic modules (#7268 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-08-29 12:36:30 +08:00
Richard Huo	ce580ce4f5	[None][feat] KV Cache Connector API (#7228 ) Signed-off-by: jthomson04 <jwillthomson19@gmail.com> Signed-off-by: richardhuo-nv <rihuo@nvidia.com> Co-authored-by: jthomson04 <jwillthomson19@gmail.com> Co-authored-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>	2025-08-28 23:09:27 -04:00
Shiyu Li	b093d94d34	[https://nvbugs/5445466 ][fix] Bypass MLP TP split for MNNVL in DeepSeek V3 to avoid hanging. (#6886 ) Signed-off-by: Shiyu Li <shili@nvidia.com>	2025-08-28 15:17:48 -07:00
dongfengy	367ff88a5e	[None][feat] Refactor llama4 for multimodal encoder IFB (#6844 ) Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>	2025-08-28 13:22:19 -07:00
Nikita Korobov	a419b77fb5	[None][fix] mxfp4 padding bug for TRT-LLM and CUTLASS MoE backends (#7214 ) Signed-off-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com>	2025-08-28 10:08:05 -07:00
Zongfei Jing	53163bf1df	[TRTLLM-6876][feat] Add low precision all2all for mnnvl (#7155 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-08-28 18:26:16 +08:00
Pengyun Lin	c1e7fb9042	[TRTLLM-7207][feat] Chat completions API for gpt-oss (#7261 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-08-28 10:22:06 +08:00
Mike Iovine	8b216135f0	[None][refactor] Move draft token padding out of Drafter (#7134 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-27 11:07:50 +02:00
dongxuy04	abdb2735be	[None][fix] Fix possible hang issue in WideEP and move some tests to pre-merge (#7262 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-08-27 01:39:24 -04:00
Yukun He	bed5bc9f2e	[None][chore] Wrap the swiglu into custom op to avoid redundant device copy. (#7021 ) A redundant D2D copy is observed when enabling torch.compile for the Llama model due to the swiglu triton kernel, which brings perf overhead. Use a custom op to wrap the swiglu op to avoid this overhead. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-27 13:02:10 +08:00
Iman Tabrizian	bc84758626	[None][feat] Add logging for OAI disagg server (#7232 )	2025-08-26 21:02:03 -07:00
Shunkangz	ff4047414b	[None][opt] Balance the request based on number of tokens in AttentionDP (#7183 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-08-27 11:16:12 +08:00
Fanrong Li	e12868bc00	[None][fix] Remove and fuse some element-wise ops in the ds-r1-fp8 model (#7238 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-08-27 10:35:38 +08:00
Jin Li	028235404b	[TRTLLM-6633][feat] Padding for piecewise cudagraph (#6750 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-08-26 18:31:33 -04:00
Fridah-nv	0f947c64cb	[None][doc] Update autodeploy README.md, deprecate lm_eval in examples folder (#7233 ) Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-08-26 10:47:57 -07:00
Frank	78ecfbb4a4	[None][fix] Fix data type of KV Cache percentage in bench. (#7230 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-08-26 12:28:09 -04:00
Void	040f4c70d3	[None][perf] Accelerate global scale calculations for deepEP fp4 combine (#7126 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-08-27 00:13:13 +08:00
Zheng Duan	cf50ba2980	[TRTLLM-6549][feat] add perf metrics endpoint to openai server and openai disagg server (#6985 ) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>	2025-08-26 15:34:44 +08:00
qixiang-99	b165f8bc97	fix/improve kvcache allocation in PyTorch runtime (#5933 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-08-26 12:40:22 +08:00
William Zhang	92576488d3	[None][feat] Skip prefetching consolidated safetensors when appropriate (#7013 ) * Why? Some models (e.g. anything produced by Mistral) can have both sharded safetensors and a consolidated safetensor in the same checkpoint directory. In such cases, prefetching both to memory is a waste of time, and memory. * What? This commit skips over consolidated safetensors when they are not the only safetensor file present in the checkpoint directory Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-08-25 23:56:21 -04:00
Leslie Fang	20922b7d1f	[None][chore] Create PyExecutor from TorchLlmArgs Part 1 (#7105 ) Signed-off-by: leslie-fang25 <leslief@nvidia.com>	2025-08-26 10:42:01 +08:00
Grzegorz Kwasniewski	2101d46d68	[TRTLLM-6342][feat] TP Sharding read from the model config (#6972 ) Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-08-25 15:41:27 -07:00
Lucas Liebenwein	97d550b4ba	[None] [AutoDeploy] canonicalize_graph before shape prop for consistent state_dict (#7223 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-08-25 16:59:57 -04:00
Bo Li	bf1b958f1a	[TRTLLM-7319][perf] Fuse slicing into MoE. (#6728 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> Signed-off-by: Sergey Klevtsov <sklevtsov@nvidia.com> Co-authored-by: Sergey Klevtsov <sklevtsov@nvidia.com>	2025-08-25 16:52:30 -04:00
Daniel Cámpora	e8e7e52892	[None][chore] Refactored the handle logits pp communication (#7154 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-08-25 16:14:08 -04:00
Frank	788fc62d23	[None][fix] Update to pull LLM from a central location. (#6458 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-08-25 13:07:29 -07:00
QI JUN	bea5e07fb7	[None][refactor] refactor the CUDA graph runner to manage all CUDA graphs (#6846 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-08-25 20:52:05 +08:00
shaharmor98	b32e00e9fd	[None][chore] remove CLI support for mamba cache dtype setting (#7119 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-08-25 08:08:51 -04:00
amitz-nv	a1e03af0f4	[TRTLLM-7346][fix] Improve performance of PyTorchModelEngine._get_lora_params_from_requests (#7033 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-08-25 10:37:40 +03:00
Enwei Zhu	be6d92f09f	[None][fix] Fix MoE load balancer config loading (#7150 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-25 01:42:54 -04:00
Yukun He	9c5b464fe0	[None][feat] Apply AutoTuner to fp8_block_scale_deep_gemm to trigger JIT ahead of time. (#7113 ) Because deep_gemm.gp8_gemm_nt will trigger many JIT processes during the inference phase, we need to sweep these shapes ahead of time. Apply the AutoTuner framework to achieve this and retain the potential capability to tune the swap_ab flag. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-25 10:48:31 +08:00
ajrasane	068056677f	[None][chore] Enable auto deploy accuracy test in CI (#7179 ) Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com> Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-08-24 08:42:30 -07:00
dongxuy04	19a0ea363b	[TRTLLM-6743][feat] Optimize and refactor alltoall in WideEP (#6973 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com> Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com> Signed-off-by: Dongxu Yang <dongxuy@nvidia.com> Co-authored-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-08-24 08:15:29 -04:00
amitz-nv	35e0ae484a	[https://nvbugs/5467232 ][fix] Fix load_torch_hf_lora to override lora_config.trtllm_modules_to_hf_modules with default only when it has no value (#7132 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-08-24 15:00:24 +03:00
Grace Ho	3d54a1a521	[None] [feat] nsys profile output kernel classifier (#7020 ) Signed-off-by: Grace Ho <grho@nvidia.com>	2025-08-23 00:57:37 -04:00
Frank	81fd468fec	[None][fix] Correct KV cache percentage report out. (#7102 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-08-22 10:28:57 -07:00
Izzy Putterman	b36460d7b5	[None][feat] Deepseek: Start Eagle work (#6210 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com> Co-authored-by: Mike Iovine <miovine@nvidia.com>	2025-08-22 12:57:17 -04:00
tomeras91	c232ba8157	[TRTLLM-4921][feat] Enable chunked prefill for Nemotron-H (#6334 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com> Signed-off-by: tomeras91 <57313761+tomeras91@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2025-08-22 12:15:20 -04:00
Suyog Gupta	e3de5758a3	[#7136 ][feat] trtllm-serve + autodeploy integration (#7141 ) Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-08-22 08:30:53 -07:00
Yiqing Yan	907bc22fcb	[None][chore] Bump version to 1.1.0rc2 (#7167 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-22 22:02:28 +08:00
Daniel Cámpora	099f081e03	[TRTLLM-7155][feat] Unify sampler handle logits implementation. (#6867 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-08-22 08:09:30 +02:00
Yukun He	983dd7e57c	[None][fix] Fix mm_placholder_counts extraction issue. (#7118 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-22 12:28:30 +08:00
Wanli Jiang	07c711eb1f	[TRTLLM-6825][fix] Update lora for phi4-mm (#6817 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-08-21 22:00:04 -04:00
dominicshanshan	6f245ec78b	[None][chore] Mass integration of release/1.0 (#6864 ) Signed-off-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com> Signed-off-by: ruodil <200874449+ruodil@users.noreply.github.com> Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Signed-off-by: Bo Deng <deemod@nvidia.com> Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: qqiao <qqiao@nvidia.com> Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> Signed-off-by: raayandhar <rdhar@nvidia.com> Co-authored-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com> Co-authored-by: ruodil <200874449+ruodil@users.noreply.github.com> Co-authored-by: Yiqing Yan <yiqingy@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com> Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Co-authored-by: Larry <197874197+LarryXFly@users.noreply.github.com> Co-authored-by: Bo Deng <deemod@nvidia.com> Co-authored-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com> Co-authored-by: Stefan Niebler <82932102+stnie@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Co-authored-by: Emma Qiao <qqiao@nvidia.com> Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com> Co-authored-by: Raayan Dhar <58057652+raayandhar@users.noreply.github.com> Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-08-22 09:25:15 +08:00
Daniel Stokes	f7c597ec40	[None][perf] Make finalize fusion part of the tactic selection logic (#6915 ) Signed-off-by: djns99 <40156487+djns99@users.noreply.github.com>	2025-08-21 14:08:03 -07:00
Fridah-nv	e18dacc931	[#4403 ][refactor] Move fusion, kvcache, and compile to modular inference optimizer (#7057 ) Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>	2025-08-21 10:30:36 -07:00
ChristinaZ	c7269ea93a	[https://nvbugs/5392414 ] [fix] Add customized default routing method (#6818 ) Signed-off-by: Christina Zhang <83400082+ChristinaZ@users.noreply.github.com>	2025-08-21 16:58:41 +08:00
Fridah-nv	647a52698a	[https://nvbugs/5443039 ][fix] Fix AutoDeploy pattern matcher for torch 2.8 (#7076 ) Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-08-21 01:14:51 -04:00
Chang Liu	75b8a90816	[None][fix] Fix llama4 multimodal by skipping request validation (#6957 ) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-08-20 21:58:53 -04:00
Yechan Kim	0893afae3d	[TRTLLM-6771][feat] Support MMMU for multimodal models (#6828 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-21 08:54:12 +08:00
Robin Kobus	b95cab2a7c	[None][ci] move unittests to sub-directories (#6635 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-20 05:42:22 -04:00
Chang Liu	ce53832610	[TRTLLM-7326][feat] Add standalone multimodal encoder (#6743 ) Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-08-19 21:42:50 -07:00
Fridah-nv	c02592d051	[None][autodeploy] Add group attention pattern for solar-pro-preview (#7054 ) Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-08-19 18:57:09 -04:00
Jinyang Yuan	0e30fe4372	[None][fix] Fix assertion errors of quantization when using online EPLB (#6922 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-08-19 11:28:36 -07:00
Michal Guzek	7334f9390c	[None][fix] Accommodate Phi3/4 to work with ModelOpt's FP8 ckpts in Torch (#6761 ) Signed-off-by: Michal Guzek <mguzek@nvidia.com>	2025-08-19 09:22:46 -07:00
zhhuang-nv	7e135d2ea7	[None][feat] Use Separate QKV Input Layout for Context MLA (#6538 ) Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com>	2025-08-19 22:04:48 +08:00
Zero Zeng	953f4fd69e	[None][fix] acceptance rate calculation fix in benchmark_serving (#6746 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-08-19 17:29:36 +08:00
Shunkangz	54ec2c1af1	[None][opt] Add batch wait timeout in fetching requests (#6923 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-08-19 03:50:08 -04:00
Yi Zhang	a15af879ec	[None][refactor] Refactor Torch Compile Backend, MoeLoadBalancer and warmup Logic (#6615 ) Signed-off-by: yizhang-nv <187001205+yizhang-nv@users.noreply.github.com> Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>	2025-08-19 09:58:44 +08:00
Daniel Cámpora	d16af87d03	[TRTLLM-7158][feat] Introduce sampler options in trtllm bench (#6855 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-08-18 18:10:05 -04:00
Kaiyu Xie	e88cb92f24	[None] [feat] Support accurate device iter time (#6906 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-08-18 13:47:14 +08:00
bhsueh_NV	85cbd0263b	[None][feat] Support Yarn on Qwen3 (#6785 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-08-17 07:21:29 +08:00
Izzy Putterman	f6ff0e3311	[None][fix] Skip Topk if 0 (#6934 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-08-16 02:17:36 -04:00
Daniel Cámpora	53312eeebd	[TRTLLM-7157][feat] BREAKING CHANGE Introduce sampler_type, detect sampler according to options (#6831 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-08-16 00:27:24 -04:00
Yiqing Yan	ec3d9f8052	[None][chore] Bump version to 1.1.0rc1 (#6953 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-16 10:32:47 +08:00
Yuening Li	1f8ae2b2db	[TRTLLM-5863][feat] Support MoE INT8 Weight-Only-Quantization in PyTorch Workflow (#6629 ) Signed-off-by: Yuening Li <62227368+yueningl@users.noreply.github.com>	2025-08-15 17:15:49 -04:00
dongfengy	0ad0b967bb	[None][fix] Make TP working for Triton MOE (in additional to EP we are using) (#6722 ) Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>	2025-08-15 16:58:42 -04:00
ajrasane	4162d2d746	[None][test] Add accuracy evaluation for AutoDeploy (#6764 ) Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com> Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-08-15 13:46:09 -04:00
yifeizhang-c	4127d77678	[https://nvbugs/5394392 ][fix] Enlarge scheduler capacity under disagg bs == 1 (#6537 ) Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>	2025-08-15 09:52:06 -07:00
liji-nv	18ccd053d3	[https://nvbugs/5427801 ][fix] Torch compile support for Llama4 and Ea… (#6858 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-08-15 11:14:20 -04:00
tomeras91	f7dbc1435a	[None] [chore] Mamba cache in separate file (#6796 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-08-15 13:42:51 +03:00
Bo Li	15aabc1540	[None][fix] Fix perfect router. (#6797 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-08-14 20:09:08 -07:00
Frank	2cc59aacb3	[None][fix] Correct reporting of torch_dtype for ModelConfig class. (#6800 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-08-14 22:46:20 -04:00
qianbiao	5c2f0fd03d	[None] [feat] Add Tencent HunYuanMoEV1 model support (#5521 ) Signed-off-by: sorenwu <sorenwu@tencent.com> Co-authored-by: sorenwu <sorenwu@tencent.com> Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>	2025-08-15 06:56:44 +08:00
Mike Iovine	078e907b16	[https://nvbugs/5455651 ][fix] Make ngram use XQA attention on Blackwell (#6873 ) Signed-off-by: Michael Iovine <miovine@nvidia.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> Signed-off-by: Mike Iovine <mike.iovine7@gmail.com>	2025-08-14 18:36:19 -04:00
Bo Li	26f413ad90	[https://nvbugs/5450262 ][fix] Fix unsupported alltoall use case (#6882 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-08-14 17:46:54 -04:00
Matthias Jouanneaux	69574ad730	[TRTLLM-5966][feat] Helix: extend mapping to support different CP types (#6816 ) Signed-off-by: Matthias Jouanneaux <mjoux@nvidia.com>	2025-08-14 09:00:02 -07:00
kris1025	4aed7a7d19	[TRTLLM-6853][feat] refactor deepseekv3 model (#6698 ) Signed-off-by: linquanh <linquanh@nvidia.com>	2025-08-14 11:03:17 -04:00
Pengbo Wang @ NVIDIA	ffc976ceaf	[https://nvbugs/5445466 ][fix] fix deepseek r1 hang by not enabling mnnvl by default (#6860 ) Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com> Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com>	2025-08-14 22:36:56 +08:00
Shi Xiaowei	1095dfd03c	[None][fix] BREAKING CHANGE: Mismatch between docs and actual commands (#6323 )	2025-08-14 03:48:57 -04:00
Yan Chunwei	0132c1db84	[https://nvbugs/5427043 ][fix] request length exceeds max_num_tokens (#6821 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-08-14 13:31:12 +08:00
Bo Deng	d8acca495b	[TRTLLM-6675][infra] Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6623 (#6735 ) Signed-off-by: Bo Deng <deemod@nvidia.com>	2025-08-14 04:36:38 +00:00
jmydurant	4200fa46d1	[None][feat] Add support for Hopper MLA chunked prefill (#6655 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-08-14 10:39:26 +08:00
Izzy Putterman	ef53de8eef	[None][feat] Add test for speculative rejection sampler (2-model) (#6542 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-08-13 22:09:35 -04:00
Tin-Yin Lai	6c52bb07ff	[https://nvbugs/5302040 ][feat] Add whisper support (Bert Attention on SM100 and GPTAttention for cross attention on SM100) (#5527 ) Signed-off-by: tinyinl <tinyinl@nvidia.com>	2025-08-13 11:19:13 -07:00
danielafrimi	bda42f8c3a	[None][feat] Support running heterogeneous model execution for Nemotron-H (#6866 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-08-13 19:51:19 +03:00
Anthony Chang	2198587b35	[https://nvbugs/5378031 ] [feat] Hopper W4A8 MoE supports ModelOpt ckpt for PyT backend (#6200 ) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>	2025-08-13 21:24:40 +08:00
Yukun He	bc5f766e0e	[TRTLLM-4501][feat] AutoTuner tuning config refactor and valid tactic generalization. (#6545 ) * Generalize the definition of tactics so that users can implement more customizable tactic types, making the configurations clearer for each kernel run. * Allow the user not to specify the `gen_tuning_buckets` or the `map_to_tuning_buckets` function. * Other code refactoring. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-13 16:25:22 +08:00
Void	1d80df0955	[None][feat] DeepEP LL combine FP4 (#6822 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-08-13 04:20:21 -04:00
Mike Iovine	f68e03e646	[https://nvbugs/5452167 ][fix] Fix ngram padding issue (#6837 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-13 11:23:16 +08:00
Yechan Kim	12102e2d48	[TRTLLM-6772][feat] Multimodal benchmark_serving support (#6622 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-12 19:34:02 -07:00
Fanrong Li	1bbc0e323b	[None][fix] Pre-allocate workspaces for DeepGEMM MoE to avoid frequent cudaFree/cudaMalloc (#6811 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-08-13 10:27:57 +08:00
rakib-hasan	2923eb88a1	[None][fix] Refactoring input prep to allow out-of-tree models (#6497 ) Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-08-12 20:29:10 -04:00
dongxuy04	bd9a6dd9ab	[TRTLLM-7008][fix] fix wideEP weights loading and args (#6789 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-08-12 19:14:20 -04:00
Robin Kobus	45c7518032	[None][refactor] Simplify decoder state initialization (#6559 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-12 21:44:41 +02:00
Robin Kobus	dd11e08d26	[#6187 ][feat] add LayerNorm module (#6625 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-12 21:43:30 +02:00
nvchenghaoz	81f0ded1c4	[None][feat] Add GPT OSS support for AutoDeploy (#6641 ) Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>	2025-08-12 14:03:22 -04:00
Jhao-Ting Chen	a060e12041	[https://nvbugs/5438869 ][fix] Set nvfp4 expert w1 w3 weight scale to the same value if they're not (#6656 ) Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>	2025-08-12 20:47:10 +08:00
Shunkangz	ab0d768acf	[None][fix] Fix attention dp log (#6570 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-08-12 04:53:09 -04:00
Liao Lanyu	f7c13a4aa7	[TRTLLM-6906][chore] Using pybind to bind functions in thop/attentionOp (#6745 ) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>	2025-08-12 16:45:16 +08:00
Sergey Klevtsov	27fc35175e	[None][feat] CUTLASS MoE FC2+Finalize fusion (#3294 ) Signed-off-by: Sergey Klevtsov <sklevtsov@nvidia.com>	2025-08-12 15:56:48 +08:00
Fridah-nv	0dc4b4e699	[#4403 ][autodeploy] Refactor: Move more transformations to new inf optimizer, Add quantization_source to factory interface (#6760 ) Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>	2025-08-11 22:02:46 -07:00
Enwei Zhu	7c686ba8de	[TRTLLM-2285][feat] Enable guided decoding with CUDA graph padding and draft model chunked prefill (#6774 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-12 09:30:06 +08:00
Ziyi Xiong	b4fcd5f592	[https://nvbugs/5441438 ][fix] Set correct draft length for the cuda graph dummy request (#6701 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-08-12 09:28:47 +08:00
Jinyang Yuan	ead89a0e40	[None][perf] Improve the performance of online EPLB on Hopper by better overlapping (#6624 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-08-12 09:25:13 +08:00
Chang Liu	be9dd4713c	[https://nvbugs/5385987 ][fix] Fix Qwen2 quantization issue by pinning transformers version (#6673 ) Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-08-11 17:16:49 -07:00
rakib-hasan	7ab8112450	[None][fix] Refactoring to avoid circular import when importing torch models (#6720 ) Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-08-11 18:00:42 -04:00
bhsueh_NV	83dbc6c75d	[TRTLLM-5532][feat] store the block of context request into kv cache (#6683 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-08-11 16:14:52 +08:00
Tracin	49bcaa4e95	Add gpt-oss GSM8K test. (#6732 ) Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>	2025-08-10 22:45:43 -04:00
Zero Zeng	4b4b91ab51	[None][feat] improve dataloading for benchmark_dataset by using batch… (#6548 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-08-11 09:50:41 +08:00
Yechan Kim	60073a7ad9	[None][feat] Support SharedTensor on MultimodalParams (#6254 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-10 17:48:24 -07:00
shaharmor98	b6baa9ed9b	[TRTLLM-6823][doc] Add checkpoint refactor docs (#6592 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-08-10 19:47:39 -04:00
shaharmor98	14b36e07d7	[TRTLLM-6174][feat] Enable FP32 mamba ssm cache (#6574 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-08-10 16:27:51 -04:00
Gal Hubara-Agam	3c5aec19c2	[#5048 ][enhance] AutoDeploy: Optimize prepare_inputs (#6634 ) Optimize prepare_inputs routine in AutoDeploy, as part of the effort to reduce the performance gap compared to the default backend. This PR includes two major fixes, and some other minor tweaks: 1. Avoid back and forth data copies 2. Optimize position ids update by separating the implementation for generation mode and context mode. Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-08-10 13:55:04 +03:00
Ziyi Xiong	de472828b9	[TRTLLM-6637][feat] Resolve KV cache divergence issue (#6628 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-08-09 23:15:04 +08:00
Yilin Fan	d643aef73c	[Perf] Improve Llama4 performance for small max_seqlen cases (#6306 ) Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>	2025-08-09 02:58:31 -04:00
Ye Zhang	bcf5ec0c9a	[None][feat] Core Metrics Implementation (#5785 ) Signed-off-by: Ye Zhang <zhysishu@gmail.com> Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-08-09 02:48:53 -04:00
Yibin Li	97787883c3	[TRTLLM-6420][feat] add support for Eclairv2 model - cherry-pick changes and minor fix (#6493 ) Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>	2025-08-08 21:40:48 -04:00
dongfengy	d06675071e	[None][fix] WAR GPT OSS on H20 with Triton MOE (#6721 ) Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>	2025-08-08 19:47:09 -04:00
Mike Iovine	90145cf557	[None][feat] Optimize CUDA graph memory usage for spec decode cases (#6718 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-08 13:56:53 -04:00
Wanli Jiang	d45236b253	[TRTLLM-6308][feat] Support Aggregate mode for phi4-mm (#6184 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-08-08 20:09:26 +08:00
Stefan Niebler	b8f036f264	[TRTLLM-6650][fix] Enhance CUDA graph + Beam search to correctly handle padding (#6665 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-08-08 14:00:33 +02:00
Liao Lanyu	32ad7f3c12	[None][fix] Remove lock related typo in py_executor (#6653 ) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>	2025-08-08 17:48:57 +08:00
JunyiXu-nv	5f45227a93	[https://nvbugs/5437106 ][fix] Fix llama4 scout TRTLLM attn_backend (#6690 ) Signed-off-by: Junyi Xu <junyix@nvidia.com>	2025-08-08 17:48:23 +08:00
Yuxian Qiu	9ff4e75f14	[None][refactor] Combine resmooth_to_fp8_e8m0 and transform_sf_into_required_layout (#6654 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-08-08 17:11:41 +08:00
Li Min	d913955952	[TRTLLM-6898][feat] make fused_moe_cute_dsl work on blackwell (#6616 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-08-08 15:03:48 +08:00
2ez4bz	064eb7a70f	[TRTLLM-5252][fix] Propagate mapping to intermediate layers (#6611 ) This commit propagates the mapping to intermediate layers to enable tensor parallelism (amongst other things) in them. It also fixes issues with a unit test for TP for pixtral, and adds it to a test list. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-08-08 01:50:36 -04:00
Enwei Zhu	aee828d98a	[TRTLLM-6854][feat] Enable guided decoding with disagg serving (#6704 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-08 12:10:36 +08:00
zhanghaotong	1cf669496a	[None][fix] Fix unnecessary GPU synchronization in torch sampler caused by incorrect tensor reference (#6626 ) Signed-off-by: 皓聪 <zhanghaotong.zht@alibaba-inc.com> Co-authored-by: 皓聪 <zhanghaotong.zht@alibaba-inc.com>	2025-08-07 23:44:47 -04:00
NVJiangShao	2f2f5cc72c	[TRTLLM-6744][feat] Remove input_sf swizzle for module WideEPMoE (#6231 ) Signed-off-by: Jiang Shao <91270701+StudyingShao@users.noreply.github.com>	2025-08-08 11:13:42 +08:00
Daniel Cámpora	efca359b66	[TRTLLM-6785][feat] BREAKING CHANGE Enable TRTLLM sampler by default (#6216 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-08-07 22:19:37 -04:00
Iman Tabrizian	82276167e6	[None][feat] Add NCCL Symmetric Integration for All Reduce (#4500 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-08-07 17:28:14 -07:00
Haohang Huang	980929e1a9	[https://nvbugs/5410687 ][fix] Hopper w4a8 groupwise MoE interleave (#6708 ) Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-08-07 15:30:16 -07:00
Yuan Tong	db8dc97b7b	[None][fix] Migrate to new cuda binding package name (#6700 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-08-07 16:29:55 -04:00
Mike Iovine	e968f98b43	[None][feat] Clean up ngram auto mode, add max_concurrency to configs (#6676 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-07 12:51:47 -04:00
Emma Qiao	3c44b44e45	[None][infra] Fix guardwords (#6711 ) Signed-off-by: qqiao <qqiao@nvidia.com>	2025-08-07 21:06:47 +08:00
pcastonguay	453a06e6ab	[TRTLLM-6881][feat] Include attention dp rank info with KV cache events (#6563 ) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>	2025-08-07 14:17:07 +02:00
Enwei Zhu	1b9781e8e7	[TRTLLM-6409][feat] Enable guided decoding with speculative decoding (part 1: two-model engine) (#6300 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-07 05:53:48 -04:00
hlu1	8207d5fd39	[None] [feat] Add model gpt-oss (#6645 ) Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>	2025-08-07 03:04:18 -04:00
amitz-nv	85af62184b	[TRTLLM-6683][feat] Support LoRA reload CPU cache evicted adapter (#6510 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-08-07 09:05:36 +03:00
Yiqing Yan	5fa1914cab	[None][chore] Bump version to 1.1.0rc0 (#6651 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-07 13:39:49 +08:00
Izzy Putterman	7e0158b583	Qwen3: Fix eagle hidden states (#6199 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-08-06 17:05:18 -04:00
Hanjun Cho	80f918cc22	[None][feat] Add Qwen3 MoE support to TensorRT backend (#6470 ) Signed-off-by: gkswns0531 <gkswns0531@gmail.com> Signed-off-by: hanjuncho <gkswns0531@gmail.com> Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>	2025-08-06 17:02:35 +08:00
Zongfei Jing	0ff8df95b7	[https://nvbugs/5433581 ][fix] DeepGEMM installation on SBSA (#6588 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-08-06 16:44:21 +08:00
Netanel Haber	83ee91e17b	[None][fix] Fix 6522 mpi.pkl5.intracomm.Request has wait not Wait (#6646 ) Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-08-06 14:18:09 +08:00
JunyiXu-nv	13e0214fe0	[TRTLLM-6263][feat] Enable fp8 SwiGLU to minimize host overhead (#6540 ) Signed-off-by: Junyi Xu <junyix@nvidia.com>	2025-08-06 10:42:19 +08:00
brb-nv	9a01934dbf	[None][feat] Switch to internal version of MMProjector in Gemma3 (#6572 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-08-05 21:48:23 -04:00
yunruis	3ff4f503ad	[None][opt] ADP schedule balance optimization (#6061 ) Signed-off-by: yunruis <205571022+yunruis@users.noreply.github.com>	2025-08-06 09:38:02 +08:00
Yechan Kim	c17f4984e2	[None][feat] Refactor Llava-Next (#6478 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-05 17:53:53 -07:00
Aurelien Chartier	6da95f29a9	[None][feat] Add support for fused gate_up_proj scales for FP8 blockwise (#6496 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-08-05 11:22:32 -07:00
Wanli Jiang	46df8712c8	[https://nvbugs/5355007 ][fix] Set `enable_chunked_context` as True by default in trtllm bench (#6582 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-08-05 11:11:36 -07:00
ixlmar	1ebceb790d	[TRTLLM-5508][feat] check input tokens + improve error handling (#5170 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-08-05 18:27:43 +01:00
liji-nv	dcbfa7e509	[https://nvbugs/5252313 ][fix] Fix torch compile + MTP (#6554 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-08-05 10:31:29 -04:00
Venky	61da2daeb4	[TRTLLM-6761][refactor] Replace LogitBiasLogitsProcessor with embedding bias tensor system (#6464 ) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>	2025-08-05 07:14:24 -07:00
Pengbo Wang @ NVIDIA	c289880afb	[None][fix] fix kimi k2 serving and add test for Kimi-K2 (#6589 ) Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>	2025-08-05 18:05:33 +08:00
amitz-nv	dc84695520	[TRTLLM-6826][feat] Allow sending more than 2GiB through MPI by using mpi4py.util.pkl5 (#6522 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-08-05 11:28:26 +03:00
danielafrimi	ed801ff74b	[None][fix] Remove expand configuration from mamba2 mixer (#6521 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-08-05 04:18:25 -04:00
Haohang Huang	c9eebcb454	[TRTLLM-6674][feat] (Breaking Change) Hopper SWA non-cyclic kernels + KV reuse + Spec Dec (#6379 ) Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com> Signed-off-by: symphonylyh <31998628+symphonylyh@users.noreply.github.com>	2025-08-05 07:47:41 +00:00
kris1025	6a3a921284	[TRTLLM-6685][feat] Add speculative metrics for trt llm bench (#6476 ) Signed-off-by: linquanh <linquanh@nvidia.com>	2025-08-04 15:22:57 -07:00
Olya Kozlova	13cc1c4878	[TRTLLM-5271][feat] best_of/n for pytorch workflow (#5997 ) Signed-off-by: Olya Kozlova <okozlova@nvidia.com>	2025-08-04 14:08:06 +02:00
brb-nv	87e4e9f468	[None][chore] Add unit test for Gemma3 lora (#6560 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-08-04 04:56:57 -04:00
Yiqing Yan	3916dbd98b	[None][chore] Bump version to 1.0.0rc6 (#6597 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-04 04:39:15 -04:00
Pengyun Lin	a15e33351d	[None][fix] Revert commit `48ddc3d` & add test for disagg server with different max_num_tokens (#6259 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-08-04 15:09:51 +08:00
Yuan Tong	a2f271c8e0	[TRTLLM-4406][feat] LLM sleep & wakeup Part 1: virtual device memory (#5034 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-08-04 13:51:01 +08:00
Yechan Kim	ee6ab5be96	chore: add EXAONE4 accuracy test (#6397 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-04 10:14:16 +08:00
Jinyang Yuan	df90202b51	[fix] Fix DeepSeek w4a8 weight loading (#6498 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-08-04 10:12:06 +08:00
Chuang Zhu	542f552d0b	use cudaSetDevice to create context ,fix nvbug 5394497 (#6403 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-08-03 13:32:55 -04:00
Shunkangz	67a3fd858b	[None][feat] Add support of scheduling attention dp request (#6246 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Co-authored-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>	2025-08-01 20:38:01 -04:00
Richard Huo	31802de0b0	[None][fix] Serialize the window_size in the kv event (#6526 ) Signed-off-by: richardhuo-nv <rihuo@nvidia.com>	2025-08-01 15:25:18 -07:00
Lucas Liebenwein	5247df6ae2	[AutoDeploy] merge feat/ad-2025-07-22 (#6520 ) Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Gal Agam <ghubaraagam@cw-dfw-cs-001-login-01.cm.cluster> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: haoguo <67671475+h-guo18@users.noreply.github.com> Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Co-authored-by: Gal Agam <ghubaraagam@cw-dfw-h100-004-328-012.cm.cluster> Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Co-authored-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>	2025-08-01 08:51:08 -07:00
brb-nv	7447d6ed85	[TRTLLM-6657][feat] Add LoRA support for Gemma3 (#6371 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-08-01 09:19:54 -04:00
liji-nv	1daa8c3232	[https://nvbugs/5340941 ][https://nvbugs/5375785 ] - fix: Wrap attentio… (#6355 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-08-01 07:38:06 -04:00
Yukun He	90856bf97d	[https://nvbugs/5419069 ][fix] Fix the mismatched layer name components. (#6417 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-01 16:32:39 +08:00
Zero Zeng	48768fd720	fix: Fix missing key (#6471 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-08-01 14:25:58 +08:00
Robin Kobus	d3c14682f0	refactor: Remove unused buffers and bindings from sampler (#6484 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-01 00:43:03 -04:00
Jaedeok Kim	fbee279909	fix: remove duplicate layer multiplication in KV cache size calculation (#6481 ) Signed-off-by: Jaedeok Kim <jaedeokk@nvidia.com>	2025-07-31 22:34:34 -04:00
Zongfei Jing	7bb0a78631	Deepseek R1 FP8 Support on Blackwell (#6486 ) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> Co-authored-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-08-01 10:26:28 +08:00
Venky	8c165fd27a	[TRTLLM-6611][feat] Add warnings and stricter validation to LoraManager adapter loading (#6453 ) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>	2025-07-31 22:22:51 -04:00
Yukun He	00059de380	chore: Improve the AutoTuner log information. (#6368 ) * Change the fallback alert from DEBUG to WARNING level and only do it once. * Add debug information for profiling cache right after the warmup phase. * Change the level of exception message during tactic profiling from ERROR to WARNING level. All exception details are pushed to the DEBUG level. * Other trivial refinements and cleanups. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-01 09:19:52 +08:00
brb-nv	2eca0d5925	fix: Fix poor generation with FP8 Gemma3 1B checkpoint (#6499 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-31 17:18:23 -07:00
Simeng Liu	8cf3faa26a	[feat] Auto-enable ngram with concurrency <= 32. (#6232 ) Signed-off-by: Simeng Liu <simengl@nvidia.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> Signed-off-by: Mike Iovine <mike.iovine7@gmail.com> Co-authored-by: Mike Iovine <miovine@nvidia.com> Co-authored-by: Mike Iovine <mike.iovine7@gmail.com>	2025-07-31 18:45:51 -04:00
Ziyi Xiong	8062e0fe7c	[TRTLLM-6392][feat] Support turning on/off spec decoding dynamically (#6363 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-31 15:31:39 -04:00
shaharmor98	0c42f54a39	Bugfix/fix nemotron nas lora support (#6380 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-07-31 13:39:35 -04:00
amitz-nv	1ee7a08d2b	[5830][feat] Improve LoRA cache memory control (#6220 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-07-31 09:26:38 +03:00
dongjiyingdjy	17e0d0fb1a	fix: fix illeagel memory access (#6437 ) Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>	2025-07-31 10:01:34 +08:00
Enwei Zhu	4b299cb77e	feat: Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 (#6408 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-31 09:53:52 +08:00
Vadim Gimpelson	25cd4f215e	[PERF] Move calculation Qwen2-VL's rotary_cos_sin to LLM worker process (#6004 ) Signed-off-by: Vadim Gimpelson <vadim.gimpelson@centml.ai>	2025-07-31 09:35:24 +09:00
shaharmor98	f9cf683e39	add propagation of trust_remote_code to OpenAIServer (#6446 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-07-30 15:25:41 -04:00
Wanli Jiang	9632dba02e	feat: TRTLLM-6450 update long rope for phi3.5/phi4-mini/phi4-mm (#6353 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-30 09:20:16 -07:00
NVShreyas	e67f4da9b5	[Perf]: Add residual, norm for nemotron_nas models (#6455 ) Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>	2025-07-30 09:10:38 -07:00
Chang Liu	b4065d8ca6	[TRTLLM-6654][feat] Add support for external multimodal embeddings (#6263 ) Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>	2025-07-30 10:00:15 -04:00
pcastonguay	e7ae5e2824	feat: Add support for disaggregation with pp with pytorch backend (#6369 ) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> Signed-off-by: raayandhar <rdhar@nvidia.com> Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> Signed-off-by: pcastonguay <55748270+pcastonguay@users.noreply.github.com> Co-authored-by: raayandhar <rdhar@nvidia.com> Co-authored-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2025-07-30 09:42:13 -04:00
tomeras91	a2514d93fc	[nvbug 5380101][fix] Fix nemotronNAS loading for TP>1 (#6447 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-07-30 07:22:32 -04:00
QI JUN	2fe9cc0889	chore: remove draft_model_engine from init parameter list of PyExecutor (#6325 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-30 03:31:49 -04:00
QI JUN	1f39a11af0	chore: clean code of PyExecutor (#6445 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-30 02:11:43 -04:00
2ez4bz	d6eed1b624	[fix] Switch placement of image placeholder for mistral 3.1 (#6435 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-30 14:10:36 +08:00
Jinyang Yuan	a427f5bece	[fix] Fix wide EP when using DeepEP with online EPLB (#6429 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-07-30 00:13:18 -04:00
Zheng Duan	c9ed1ab436	[TRTLLM-6549] chore: record delay introduced by disaggregated serving in kv cache measure (#6135 ) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>	2025-07-30 10:39:40 +08:00
peaceh-nv	5b420ad267	Rename layer to comply with deepseek (#6393 ) Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>	2025-07-30 10:00:48 +08:00
Yechan Kim	d6eb8e2366	fix: support mixture of text & multimodal prompts (#6345 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-30 08:52:31 +08:00
Yunfan Fan	1a8e28d295	[FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine (#6344 ) Signed-off-by: fanyunfan <2569548856@qq.com> Co-authored-by: fanyunfan <2569658856@qq.com>	2025-07-30 07:13:44 +08:00
Yan Chunwei	ad662ddcdd	chore: disallow arbitrary in llm_args.Configs (#6367 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-29 16:16:52 -04:00
Michal Guzek	7efe3cb0cd	[fix] Add detokenization-based stop word logic to LLM API (#5948 ) Signed-off-by: moraxu <mguzek@nvidia.com> Signed-off-by: Michal Guzek <mguzek@nvidia.com>	2025-07-29 10:16:59 -07:00
Yukun He	0eee2e2850	[5385981] fix: Update the usage of VisionAttention init API. (#6413 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-07-29 16:41:48 +08:00
QI JUN	13e24ab1cb	chore: remove unused code in PyExecutor (#6351 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-29 16:24:26 +08:00
Frank	d2a04abb95	[fix] Fixes to parameter usage and low latency configuration. (#6343 )	2025-07-29 01:36:13 -04:00
nv-guomingz	49044733e1	chore: delete useless gitkeep files. (#6400 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-28 11:38:30 -04:00
QI JUN	4efc6496b7	chore: add _prepare_and_schedule_batch function in PyExecutor (#6365 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-28 05:50:27 -04:00
Yan Chunwei	45d441e60c	[TRTLLM-5061] chore: add status tags to LLM API reference (#5707 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-28 15:57:07 +08:00
Zero Zeng	c9b8b6180f	Add Acceptance Rate calculation to benchmark_serving (#6240 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-07-28 14:00:58 +08:00
Jinyang Yuan	97f7e12588	[fix] Fix perf regression caused by MoE autotuner when using DeepEPLowLatency (#6288 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-07-28 01:37:11 -04:00
Chang Liu	dc757799e1	[nvbugs/5401156][fix] Avoid import all models when import trtllm._common (#6266 )	2025-07-27 23:29:21 -04:00
Void	f172face98	DeepEP LL dispatch FP4 (#6296 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-07-28 11:25:42 +08:00
Yukun He	93a0fd0a23	[TRTLLM-6445] feat: Enable AllReduce-associated fusion patterns in Llama3/4. (#6205 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-07-28 09:36:26 +08:00
YueWeng	2dd3186727	fix: remove cudaStreamSynchronize when using relaxed acceptance (#5262 ) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>	2025-07-28 09:18:41 +08:00
Ziyi Xiong	d853811190	[https://nvbugs/5402719 ][fix]: Add cuda graph dummy requests to the spec_resource_manager (#6258 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-26 20:32:39 -04:00
Michal Guzek	08d57123f9	[nvbug/5374773] chore: Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache (#5974 ) Signed-off-by: moraxu <mguzek@nvidia.com>	2025-07-25 18:10:40 -04:00
ameynaik-hub	1e5e71aa42	Mtp optimizations round1 (#5689 ) Signed-off-by: Amey Naik <212485788+ameynaik-hub@users.noreply.github.com> Co-authored-by: Kefeng-Duan <176893526+Kefeng-Duan@users.noreply.github.com>	2025-07-25 13:48:27 -04:00
nv-guomingz	b8d4cb8beb	feat: Support JSON Schema in OpenAI-Compatible API (#6321 ) Signed-off-by: noiji <52301388+noiji@users.noreply.github.com>	2025-07-25 12:55:56 -04:00
xiaoqi	a0aecf0476	[feat]: support logit_bias (#5354 ) Signed-off-by: xq25478 <xq25478@qq.com> Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> Signed-off-by: hexiao.xq <hexiao.xq@antgroup.com> Co-authored-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> Co-authored-by: hexiao.xq <hexiao.xq@antgroup.com> Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-25 09:37:41 +00:00
liji-nv	e07fff4f78	[https://nvbugs/5340941 ] - fix: Correct custom ops used by Qwen3 Moe … (#6285 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-25 14:49:45 +08:00
Mike Iovine	0f2f11f90b	[TRTLLM-6453][feat] Support chunked prefill on spec decode 2 model (#6104 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-24 21:50:11 -04:00
Linda	9a99e6d6d7	fix: integration tests with nanobind (#6326 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-07-25 09:23:20 +08:00
Shiyu Li	375f74ecb2	[fix][nvbugs/5399355] Fix Lamport buffer clear issue for MNNVL TwoShot Allreduce and add FP16 support. (#6237 ) Signed-off-by: Shiyu Li <shili@nvidia.com>	2025-07-25 08:01:40 +08:00
Frank	f8f5ba65fc	[fix] Update to remove popping of KV cache and other args. (#6310 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-24 15:54:33 -04:00
Stefan Niebler	0df758ec9f	[TRTLLM-6650][feat] Enhance beam search support with CUDA graph integration (#6217 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-24 18:04:41 +02:00
bhsueh_NV	7b6aadc800	[Fix][nvbug 5401163][nvbug 5404726][Qwen3] Fix bug of MoE on tp > 1 with trtllm moe backend (#6235 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-07-24 21:47:37 +08:00
liji-nv	14d94a3856	feat: Add non UB AR + Residual + Norm + Quant fusion (#6320 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-24 05:51:43 -04:00
Lizhi Zhou	a63a1ac7f9	[TRTLLM-6444] Add some UCX trouble shooting docs and print UCX related logs (#6085 ) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>	2025-07-24 16:21:01 +08:00
QI JUN	428e34080f	chore: remove unused variables in pyexecutor (#6280 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-24 13:16:15 +08:00
Stefan Niebler	2486eb778e	[TRTLLM-6651][feat] Enable Overlap scheduler + Beam Search in TRTLLM Sampler (#6223 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-23 12:30:50 +02:00
YueWeng	ed62a06eef	[nvbug/5322354] fix PD + MTP + overlap scheduler accuracy issue (#6136 ) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>	2025-07-23 14:53:37 +08:00
QI JUN	a8253b942f	chore: remove duplicate should_stop_processing check (#6242 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-23 14:11:23 +08:00
Yechan Kim	83c3ed128b	chore: set default device to cpu on Multimodal models (#5994 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-22 21:45:31 -07:00
Erin	5636c67388	fix: nvbug_5398806 (#6239 )	2025-07-23 11:45:11 +08:00
Venky	9538c8d0e5	Add basic Nemo Ckpt Lora Loading in pytorch flow (#6019 )	2025-07-22 19:42:45 -07:00
wili	8ecdeee300	[refactor] Simplification of Speculative decoding configs - Part 2 (#5936 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-07-23 09:20:27 +08:00
Lucas Liebenwein	41fb8aa8b1	[AutoDeploy] merge feat/ad-2025-07-07 (#6196 ) Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com> Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com> Co-authored-by: Neta Zmora <nzmora@nvidia.com> Co-authored-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com>	2025-07-23 05:11:04 +08:00
2ez4bz	ab7434ac62	[feat] Enable TP and batching for PixtralVisionModel / Mistral3VLM (#6152 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-22 11:06:41 -07:00
John Calderon	b7c8a672da	[Issue 6193] Fix gemma3vl weight loader (#6233 ) Signed-off-by: John Calderon <johncalesp@gmail.com>	2025-07-22 10:32:18 -07:00
danielafrimi	ff9963978a	Add register_fake for finegrained_mixed_dtype_gemm torch_op (#6255 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-07-22 16:59:55 +03:00
Yiqing Yan	3e18ee5fe1	chore: bump version to 1.0.0rc5 (#6252 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-22 16:24:28 +08:00
Pengyun Lin	48ddc3d4b9	[fix]: Revert commit `388b491` (#6143 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-22 12:48:00 +08:00
Yi Zhang	eb7d0f84b5	[nvbugs/5368410][fix] Disable moe allreduce for multi node (#5918 ) Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>	2025-07-22 12:48:00 +08:00
Fanrong Li	c66941036f	fix: fix index out of bounds error in spec decoding (#5954 )	2025-07-22 12:48:00 +08:00
Shunkangz	ee45e0c63f	feat: Refactor the fetching request logic (#5786 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-07-22 09:16:28 +08:00
Chang Liu	7381f1dba7	[TRTLLM-5059][feat] Add KV cache reuse support for multimodal models (#5444 ) Only supports qwen in this PR	2025-07-21 16:11:58 -07:00
Ziyi Xiong	d7f0b0ab68	[fix] Correct the returned value of has_spec_drafter (#6178 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-21 11:38:59 -04:00
Pengyun Lin	9832bef07d	[BREAKING CHANGE]: change default backend to PyTorch in trtllm-serve (#5717 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-21 21:09:43 +08:00
liji-nv	3e0fb60e50	[TRTLLM-4279] feat: Multistream initial support for torch compile flow (#5847 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-21 19:10:22 +08:00
Linda	3efad2e58c	feat: nanobind bindings (#6185 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-07-21 08:56:57 +01:00
Yuening Li	e8c068b4b1	[TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow (#5850 ) Signed-off-by: Yuening Li <62227368+yueningl@users.noreply.github.com> Co-authored-by: Yuening Li <62227368+yueningl@users.noreply.github.com>	2025-07-21 15:17:35 +08:00
Jinyang Yuan	88076eecd0	[fix] Fix can_use_alltoall in fused_moe_wide_ep.py (#6173 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-07-21 10:53:07 +08:00
brb-nv	ca9bc5727e	fix: Flush stale `PlanParams` with custom attention mask (#6163 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-21 09:55:09 +08:00
brb-nv	a433ebad2b	enh: Lift expectation of single image per sample in Gemma3 VLM (#6195 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-21 08:43:07 +08:00
danielafrimi	5300a99bd8	W4A8 GEMM (#6005 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-07-20 17:34:57 +03:00
amitz-nv	98428f330e	[TRTLLM-5826][feat] Support pytorch LoRA adapter eviction (#5616 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-07-20 08:00:14 +03:00
Void	118307c224	DeepEP LL support variable hidden size and tokens num (#6141 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-07-20 09:32:41 +08:00
Pengyun Lin	69e9f6d489	[fix]: Skip prompt length checking for generation only requests (#6146 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-19 21:26:37 +08:00
Ziyi Xiong	66030ef815	[TRTLLM-6452][feat]: Two-model engine KV cache reuse support (#6133 ) Signed-off-by: ziyixiong-nv <fxiong@nvidia.com> Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-19 13:17:15 +08:00
Rashid Kaleem	152e2df43b	[Disaggregated] Add retry knobs and handling (#5808 ) Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com> Signed-off-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>	2025-07-19 07:27:59 +08:00
John Calderon	fc8b29c4ff	[Issue 5927][fix] Avoid memory calls during broadcast for single GPU (#6010 ) Signed-off-by: John Calderon <johncalesp@gmail.com>	2025-07-18 14:21:03 -07:00
Netanel Haber	d9a3530048	[nvbug/5393888][nvbug/5393042] Always use `py_seq_slot` (#6147 ) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-07-18 22:45:16 +03:00
Stefan Niebler	6d7874a467	[nvbugs/5369799] fix: Update disaggregation handling in sampler (#5762 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-19 01:40:46 +08:00
xiaoqi	28858c8711	feat(eagle3):support qwen3 dense model (#5879 ) Signed-off-by: xq25478 <xq25478@qq.com>	2025-07-19 01:24:32 +08:00
Bo Li	07e8813984	feat: Remove padding in attention DP. (#6064 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-07-18 23:30:34 +08:00
Stefan Niebler	fd6ce7f20e	[ci] Speedup beam search unit tests with fixtures for LLM (#5843 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-18 22:54:49 +08:00
Erin	9522cde464	fix: NVBug 5385576 py_batch_idx issue (#6153 ) Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>	2025-07-18 22:36:43 +08:00
Robin Kobus	ec2b953e7e	refactor: Enhanced handling of decoder requests and logits within the batch manager (#6055 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-18 12:12:08 +02:00
Aurelien Chartier	812243bdd6	feat: add support for Modelopt fp8_pb_wo quantization scheme (#6106 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-07-18 10:35:12 +08:00
Zhenhuan Chen	992b273045	[https://nvbugs/5387375 ] fix(scaffolding): fix scaffolding aime test in test_e2e (#6140 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-18 10:34:37 +08:00
yifeizhang-c	0155e7a3a1	[TRTLLM-6368] Update deepep dispatch API (#6037 ) Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>	2025-07-18 10:13:31 +08:00
Iman Tabrizian	b75e53ab69	Revert "feat: nanobind bindings (#5961 )" (#6160 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-18 10:12:54 +08:00
qixiang-99	2c90203c36	Refactor KVCacheManager: Simplify token availability calculation and … (#6134 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-07-17 13:33:33 -07:00
Frank	161490f039	[fix] Fixes KV Cache overrides in trtllm-bench (#6103 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-18 03:44:44 +08:00
2ez4bz	8480c120b1	[fix] Fix Mistral3VLM weight-loading & enable in pre-merge (#6105 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-17 11:04:17 -07:00
Iman Tabrizian	10dbf4f0f4	[fix] Remove duplicated KVCache transmission check (#6022 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-17 12:02:19 -04:00
Linda	5bff317abf	feat: nanobind bindings (#5961 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-07-17 22:42:52 +08:00
Ziyi Xiong	58d22a72f1	[TRTLLM-6352][feat] Migrate EAGLE3 and draft/target speculation to Drafter (#6007 ) Signed-off-by: ziyixiong-nv <fxiong@nvidia.com>	2025-07-17 21:15:01 +08:00
Enwei Zhu	21efb50068	[TRTLLM-6406] feat: Enable guided decoding with overlap scheduler (#6000 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-17 17:46:10 +08:00
Chuang Zhu	44c70c88f9	chore:[BREAKING CHANGE] use cacheTransceiverConfig as knobs for disagg service (#5234 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-07-17 17:42:07 +08:00
Iman Tabrizian	d4d21a106e	[fix] Release slots with spec decode + disagg (#5975 ) (#6032 ) Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com> Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-17 12:58:18 +08:00
Shiyu Li	6e1aee6fd6	[fix] Performance Optimization for MNNVL TwoShot Kernel (#5934 ) Signed-off-by: Shiyu Li <shili@nvidia.com> Co-authored-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-07-17 10:49:51 +08:00
chenfeiz0326	fe070a0168	test: Update Llama4 Scout FP4 & FP8 accuracy tests (#5901 ) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>	2025-07-17 09:41:18 +08:00
Wanli Jiang	2d2b8bae32	feat: TRTLLM-5574 Add phi-4-multimodal pytorch-backend support (#5644 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-17 06:30:58 +08:00
qixiang-99	e09e409dfb	Fix: Enhance ModelConfig for kv cache size calculations (#5868 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-07-16 14:41:31 -07:00
Mike Iovine	fa34cb7234	[refactor] Clean up drafter/resource manager creation logic (#5805 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-16 12:45:46 -07:00
shaharmor98	e0836f9ca9	[TRTLLM-5493] Add core infrastructure to enable loading of custom checkpoint formats (#5372 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-07-17 00:50:30 +08:00
Wanli Jiang	9354114f68	fix: Update trtllm args issues with extra nested config (#5996 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-16 12:41:45 -04:00
Bo Li	fc2347eaf5	chore: Cleanup disable_fp4_allgather. (#6006 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-07-16 17:54:36 +08:00
Yan Chunwei	a02606a9e2	[TRTLLM-5530][BREAKING CHANGE] refactor: unify KvCacheConfig in LLM class for pytorch backend (#5752 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-16 16:42:59 +08:00
Yan Chunwei	7568deb2f1	[nvbug/5387226] chore: add propogation for trust_remote_code to AutoConfig (#6001 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-16 16:05:38 +08:00
Yiqing Yan	e51c541617	chore: Bump version to 1.0.0rc4 (#6086 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-16 13:02:23 +08:00
Wanli Jiang	8679a058a3	fix: Unable to load phi4-model with tp_size>1 (#5962 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-16 11:39:41 +08:00
danielafrimi	edab7532dd	feat/add latency support for trtllm bench (#3730 ) Signed-off-by: Ubuntu <dafrimi@nvidia.com> Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com> Signed-off-by: Frank <3429989+FrankD412@users.noreply.github.com> Co-authored-by: Daniel Afrimi <dafrimi@nvidia.com> Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com>	2025-07-15 13:13:49 -07:00
Fanrong Li	7a1af1c738	Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/5947 (#5989 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-07-16 01:33:12 +09:00
Xiaodong (Vincent) Huang	0523f77b36	support TRTLLM_DEEP_EP_TOKEN_LIMIT to allow run deep-ep on memory-con… (#5684 ) Signed-off-by: Vincent Huang <vincenth@nvidia.com>	2025-07-15 18:34:21 +03:00
Tailing Yuan	4a26bd6500	Fix: pad DeepEP fp4 recv tensors if empty (#6048 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-07-15 23:14:01 +09:00
MinaHuai	9ebc3ab9c4	[nvbugs/5385972][nvbugs/5387423][Fix] Minor fix for llava_next/llava_onevision (#5998 ) Signed-off-by: Mina Huai <121143971+MinaHuai@users.noreply.github.com>	2025-07-15 10:01:35 -04:00
Jaedeok Kim	ab1c54709d	fix: adjust window sizes of VSWA at torch backend (#5880 ) Signed-off-by: Jaedeok Kim <jaedeokk@nvidia.com>	2025-07-15 17:41:54 +08:00
nv-guomingz	4e4d18826f	chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… (#6003 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-15 15:50:03 +09:00
Lucas Liebenwein	e499f6c44a	[Fix] check for ImportError or ModuleNotFoundError for deep_ep_utils (#6026 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-07-15 14:31:35 +09:00
Rashid Kaleem	2ea4077993	[Model load] Fix llama min-latency model load (#5883 ) Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>	2025-07-15 09:29:19 +08:00
ixlmar	f225f5cd2e	[nvbugs-5318143] fix: restrict PyTorch memory usage to avoid OOMs (#5964 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-07-15 06:49:42 +08:00
brb-nv	f5f5be9e94	enh: Bidirectional mask with multiple images for Gemma3 (#5976 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-14 22:39:18 +08:00
brb-nv	1a2d96919c	feat: Update Gemma3 Vision Encoder (#5973 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-14 22:38:10 +08:00
Yechan Kim	63139fdcff	feat: EXAONE4.0 support (#5696 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-14 22:28:10 +09:00
Zhenhuan Chen	30608a5e6d	[https://nvbugs/5355316 ] fix: update torch.compile option to fix triton store_cubin error (#5865 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-14 17:17:30 +08:00
Robin Kobus	5a61d64b5b	[nvbugs/5345391] fix: chunked prefill + overlap scheduling (#5761 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Pengyun Lin	388b4919b8	[nvbug 5304752][fix] enhance _check_arguments to filter illegal requests for pytorch backend (#5541 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Pengyun Lin	6992616c1f	[nvbug 5004744][fix] rewrite completion API to avoid repetitive tokens (#5201 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Iman Tabrizian	c8874a7f94	[nvbug/5337601][fix] Fix disagg + speculative decoding (#5558 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Dom Brown	afaa388bee	[TRTLLM-6100] fix: Nvbug 5356427: autotuned TRTLLM Gen fp8 block scale MoE illegal memory access (#5676 ) Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
WeiHaocheng	4d8920982a	fix: set allreduce strategy to model config (#5955 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-07-14 17:59:11 +09:00
dominicshanshan	c9e7f831dc	Breaking change: perf: [TRTLLM-4662] Enable cuda graph by default (#5480 ) Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-07-14 16:42:23 +08:00
wili	cfcb97af0e	[BUG5388075][fix] Fix error in post-merge-tests (#5949 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-07-14 14:33:39 +09:00
QI JUN	ce39409530	fix cancel request logic (#5800 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-14 10:23:20 +08:00
Mike Iovine	8950223f6f	[fix] Remove SpecConfig and fix thread leak issues (#5931 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-12 21:03:24 +09:00
Enwei Zhu	bc1d4fb5da	[NvBug 5378370] fix: Fix alltoall for llama4 (apply_router_weight_on_input=True) (#5902 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-12 15:50:31 +09:00
Thor Johnsen	041f1fa513	[TRTLLM-6264] Fix flaky test_e2e.py::test_openai_lora (#5885 ) Signed-off-by: thorjohnsen <41591019+thorjohnsen@users.noreply.github.com>	2025-07-11 16:20:41 -07:00
2ez4bz	6304866ce8	[refactor] Move vision parts from processor to model for Gemma3 (#5888 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-11 15:13:51 -07:00
brb-nv	0385f89abc	test: Fix Gemma3 unit tests due to transformers upgrade (#5921 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-10 17:24:10 -07:00
Void	854655f2f7	deepEP fp4 post quant all2all dispatch (#5881 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-07-11 08:18:54 +08:00
Frank	aa4eebe973	[enhance] Add the ability to write a request timeline. (#5258 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> Signed-off-by: Frank <3429989+FrankD412@users.noreply.github.com>	2025-07-10 17:15:30 -07:00
wili	2e3cf42e03	[refactor] Simplification of Speculative decoding configs (#5639 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-07-10 11:37:30 -04:00
Kaiyu Xie	7b09a415c1	fix: Make the bench serving script compatible with different usages (#5905 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-07-10 19:36:26 +08:00
Enwei Zhu	055c4a9fe6	[NvBug 5370718, 5371538] fix: Fix incremental detokenization (#5825 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-10 16:30:00 +08:00
CarstyYou	dc32f9ae73	[fix] fix tileN cannot % 16==0 & support sm89 deepgemm bmm (#5531 ) Signed-off-by: CarstyYou <186021327+CarstyYou@users.noreply.github.com>	2025-07-10 15:16:18 +08:00
Anthony Chang	7d21b55b5a	[feat] Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE (#5723 ) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>	2025-07-10 14:06:50 +08:00
Yan Chunwei	07f6da763d	[TRTLLM-5530] chore: rename LLM.autotuner_enabled to enable_autotuner (#5876 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-10 11:31:35 +08:00
Hanjun Cho	6490a27ad7	[feat] Add TensorRT-Engine Qwen3 (dense) model support (#5650 ) Signed-off-by: Ubuntu <ubuntu@ip-10-0-20-146.us-west-2.compute.internal> Signed-off-by: Hanjun Cho <46752251+gkswns0531@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-20-146.us-west-2.compute.internal>	2025-07-10 10:26:06 +08:00
brb-nv	3209b31665	feat: Custom masking utils for Gemma3 VLM (#5853 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-10 06:18:04 +09:00
2ez4bz	87fe44fd29	feat(models): Mistral3.1 VLM pytorch backend support (#5529 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-09 13:17:40 -07:00
Chang Liu	b61a717275	[1/N][TRTLLM-5195][feat] Share PyTorch tensor between processes (#5396 )	2025-07-10 05:12:53 +09:00
Wanli Jiang	3f7cedec7c	Update transformers to 4.53.0 (#5747 ) Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com> Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-09 09:32:24 -07:00
DylanChen-NV	74dca0aa7b	[NVBUG-5304516/5319741]Qwen2.5VL FP8 support (#5029 ) Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>	2025-07-09 23:16:42 +08:00
tomeras91	5aa958a11a	[TRTLLM-5838][fix] fix max batch size and max tokens in kv cache estimations for Nemotron-H (#5371 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-07-09 11:30:15 +03:00
Dom Brown	3e3b1769ad	[TRTLLM-5881] feat: Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner (#5764 ) Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-07-09 08:21:58 +01:00
dongxuy04	dd3c736c7e	chore: some refactor on WideEP (#5727 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-07-09 14:26:57 +08:00
chenfeiz0326	64fd64fcf2	[TRTLLM-6262] Fix Llama4 Scout FP4 crash issue (#5834 ) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>	2025-07-09 14:23:21 +08:00
Chang Liu	4df5f96c8d	[Bugfix] LLama4: fix for llama4 multimodal support (#5809 )	2025-07-09 13:03:40 +09:00
Xianjie Qiao	5ab1cf5ae6	Remove unnecessary benchmarking results (#5852 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-07-09 11:19:06 +08:00
brb-nv	2bd09ed2d4	fix: Skip rope scaling for local layers in Gemma3 VLM (#5857 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-09 10:10:33 +08:00
Omer Ullman Argov	d6d2ab2c99	[fix] Catch inference failures in `trtllm-bench` (#5841 ) Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>	2025-07-09 03:53:03 +03:00
Iman Tabrizian	c508b994b6	Fix lost requests for disaggregated serving (#5815 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-09 08:42:45 +09:00
Kaiyu Xie	bb5b16fcb9	feat: Return context response immediately when stream_interval > 1 (#5836 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-07-09 00:19:57 +09:00
Raayan Dhar	e3268a4221	[TRTLLM-5847][feat] Support n-gram speculative decoding with disagg (#5732 ) Signed-off-by: raayandhar <rdhar@nvidia.com>	2025-07-08 09:39:58 -04:00
Yukun He	e104f8bbb5	[5305318] fix: Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. (#5801 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-07-08 19:51:05 +08:00
Yegor	b01d1c28f7	[feat] Detokenize option in /v1/completions request (#5382 ) Signed-off-by: Yegor <75512761+Wokzy@users.noreply.github.com> Signed-off-by: Yegor Yershov <yegor6741@gmail.com>	2025-07-08 19:36:04 +08:00
xiweny	eaf8bec88b	fix: Disaggregate serving with attention DP (#4993 ) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>	2025-07-08 16:15:03 +08:00
Yiqing Yan	5203a0f6df	chore: bump version to 1.0.0rc3 (#5819 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-08 16:04:40 +09:00
Zhenhuan Chen	dee6644ed9	feat(scaffolding): add streaming scaffolding_llm.generate_async support (#5345 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-08 15:08:40 +09:00
nv-guomingz	0be41b6524	Revert "chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie…" (#5818 )	2025-07-08 13:15:30 +09:00
Yechan Kim	5bc3a15f10	feat: add MultimodalParams & putting all multimodal params into it and refactor HyperCLOVAX & Qwen2/2.5-VL (#5522 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-07 18:03:12 -07:00
nv-guomingz	5a8173c121	chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… (#5795 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-08 08:52:36 +08:00
Robin Kobus	30a19fcf7c	[TRTLLM-6291] feat: Add user-provided speculative decoding support (#5204 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-07 16:30:43 +02:00
Tailing Yuan	85b4a6808d	Refactor: move DeepEP from Docker images to wheel building (#5534 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-07-07 22:57:03 +09:00
Daniel Cámpora	1260e2f33f	feat: Optimize TRTLLM Sampler perf single beam single step (#5550 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-07-07 15:44:47 +02:00
DylanChen-NV	5ca2b9bb15	[TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow (#5615 ) Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>	2025-07-07 18:04:57 +08:00
Yan Chunwei	dfce61f4b9	[TRTLLM-5530][BREAKING CHANGE] refactor: LLM arglist rename mixed_sampler to enable_mixed_sampler (#5751 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-07 17:05:14 +08:00
Zheng Duan	de10774c2e	chore: log stack trace on error in openai server (#5749 ) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>	2025-07-07 14:54:36 +08:00
Daniel Stokes	ec6c7dff1a	feat: Add support for MXFP8xMXFP4 in pytorch (#5535 ) Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>	2025-07-06 15:32:06 -07:00
Robin Kobus	ae27261094	refactor: decoding inputs (#5679 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-06 08:21:02 +02:00
Xianjie Qiao	b1976c2add	Add wide-ep benchmarking scripts (#5760 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> Signed-off-by: Xianjie Qiao <5410381+qiaoxj07@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-05 19:29:39 +08:00
Xianjie Qiao	089fd55eda	Add dummy all_reduce for kernel breakdown (#5745 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-07-05 13:08:58 +09:00
Frank	d61893dc77	[fix] Update to properly set cuda graphs in trtllm-bench overrides. (#5634 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-05 05:19:16 +09:00
Stefan Niebler	d1112aac37	[TRTLLM-3442] feat: added beam search support to the PyTorch Workflow (#5333 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-05 01:35:13 +09:00
HuiGao-NV	3ed3bbcb5d	Fix: pass allreduce strategy to pytorchConfig (#5746 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-07-04 21:32:13 +09:00
Shunkangz	32339d1b20	Raise shut down error for each request (#4936 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-07-04 18:58:24 +09:00
Tailing Yuan	e134a52e07	Perf: reduce DeepEPLowLatency memory and time (#5712 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-07-04 14:46:28 +08:00
Shunkangz	a79d8c9f5e	Fix none response in PD (#5422 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-07-04 14:25:10 +08:00
brb-nv	cdaa6abce7	fix: Investigate Gemma3 1B decoder output discrepancy (#5564 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-04 13:14:13 +08:00
Frank	819ae903de	[https://nvbugspro.nvidia.com/bug/5351333 ][fix] Update to chunking calculation. (#5625 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-04 13:14:13 +08:00
Clay	7a319524da	feat: support more parameters in openai worker of scaffolding (#5115 ) Signed-off-by: Clay <ccs96307@gmail.com>	2025-07-04 09:35:34 +08:00
Lucas Liebenwein	24ac9b5f69	[AutoDeploy] merge feat/ad-2025-06-29 (#5737 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Neta Zmora <nzmora@nvidia.com> Co-authored-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>	2025-07-04 10:21:18 +09:00
Netanel Haber	aa72d39b72	MTP and derivatives: Align sample state with trtllm sampler sample state (#5675 ) This PR moves MTPSampler and derivatives to use the universal seq_slot indexing for sampling. This is the last piece of the puzzle: After this, all of the samplers will use this format. See: `6ee94c7` Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-07-03 19:55:48 +02:00
Zhenhuan Chen	528ff52ef4	[https://nvbugs/5365714 ] fix(scaffolding): use default LLM rather than trt backend LLM (#5705 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-03 23:54:20 +09:00
Rashid Kaleem	2b0c87e613	[ModelLoad] Concurrent load model (#5291 ) Signed-off-by: Rashid K <rkaleem@nvidia.com> Co-authored-by: Zhihan Jiang <68881590+nvzhihanj@users.noreply.github.com>	2025-07-03 22:18:04 +08:00
nv-guomingz	8dad22cbe7	chore: refine the default value by using pydantic default instead of … (#5695 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-03 22:41:29 +09:00
tomeras91	7dbecf7272	[TRTLLM-4923][feat] Enable CUDA graphs for Nemotron-H (#5646 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-07-03 11:07:51 +03:00
Yiqing Yan	3c9dd5cd66	chore: bump version to 1.0.0rc2 (#5645 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-03 12:35:28 +08:00
Enwei Zhu	3a46cf275b	fix: Fix missing arg to alltoall_prepare_maybe_dispatch (#5669 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-02 21:41:55 -04:00
Jhao-Ting Chen	77082cde38	[https://nvbugspro.nvidia.com/bug/5329655 ] [feat] Pytorch path add spec dec param to attention op (#5146 ) Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>	2025-07-02 04:54:43 -04:00
qixiang-99	ca7b6ec8d8	Feat/pytorch vswa kvcachemanager (#5151 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-07-02 15:58:00 +08:00
Yan Chunwei	2d69b55fe8	chore: enhance yaml loading arbitrary options in LlmArgs (#5610 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-02 14:21:37 +08:00
HuiGao-NV	10c50515c2	fix: Add back allreduce_strategy parameter into TorchLlmArgs (#5637 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-07-02 09:49:20 +08:00
Perkz Zheng	ba2ab5098b	[Bug] attention DP doesn't work with embedding TP (#5642 ) Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>	2025-07-02 08:57:46 +08:00
Aurelien Chartier	efef911f5e	fix: add missing self. from PR #5346 (#5653 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-07-01 20:38:55 -04:00
Aurelien Chartier	fa95e402a5	feat: add LLmArgs option to force using dynamic quantization (#5346 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-07-01 12:16:09 -07:00
liji-nv	c345f5876c	[feat] Support torch compile for attention dp (#5086 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-01 13:48:52 -04:00
Kaiyu Xie	f9a455651b	perf: Use tokenizers API to optimize incremental detokenization perf (#5574 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-07-01 09:35:25 -04:00
Anurag Mukkara	93edfea2b8	[nvbug/5354825] Fix nougat test image url (#5496 ) Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>	2025-07-01 20:12:55 +08:00
Wanli Jiang	3789ba1d37	feat: TRTLLM-5941 Upgrade xgrammar to 0.1.18 (#5364 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-01 20:12:55 +08:00
brb-nv	4ef60d5fbb	nvbugs-5331031; nvbugs-5344203 - address intermittent issues with Mistral Small multimodal for BS=8 (#5453 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-01 20:12:55 +08:00
danielafrimi	7a617ad1fe	feat: W4A16 GEMM (#4232 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-07-01 10:36:05 +03:00
Netanel Haber	6ee94c7ac8	Reintroduce with perf fixes: feature: unify new_tokens format sample state to trtllm samper tokens format (#5513 ) `58a8a8f` - these changes were previously merged to main here. `6aef149` - the changes were temporarily reverted in main, due to a significant perf regression in models using the TorchSampler (observed by @byshiue). This PR is meant to re-merge these changes along with a fix to prevent the regression. The first commit of this PR is actually just the reverted revert - filter it out of the changes to see previously unmerged changes. Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-06-30 11:58:59 -07:00
Wei-Ming Chen	f28cd3056e	feat: AutoDeploy fp8 quantization support for bmm (#3849 ) Signed-off-by: Wei-Ming Chen <17592131+meenchen@users.noreply.github.com>	2025-06-30 12:36:34 -04:00
nv-guomingz	6e48ac25a6	chore: remove cuda_graph_ prefix from cuda_graph_config filed members. (#5585 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-06-30 12:23:14 -04:00
Li Min	16fc99391f	refactor: [TRTLLM-6150] Refactor moe permute and finalize op by removing duplicated code (#5557 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-06-30 08:48:04 -07:00
Yan Chunwei	98a7c24062	chore [TRTLLM-6009]: remove ptuning knobs from TorchLlmArgs (#5595 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-30 20:40:23 +08:00
Robin Kobus	9bdc5951f8	refactor: decoder state setup (#5093 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-30 11:09:43 +02:00
Fanrong Li	6cbc9a5297	[nvbug/5354946][fix] Fix mtp vanilla draft inputs (#5568 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-30 15:59:12 +08:00
WeiHaocheng	42a9385d02	[TRTLLM-5331] perf: Replace allgaher with AllToAllPrepare (#5570 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-06-30 13:06:09 +08:00
dongjiyingdjy	852b79053d	feat : support duplicate_kv_weight for qwen3 blockwise scale (#5459 ) Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>	2025-06-30 11:49:22 +08:00
nv-guomingz	578430e64c	[TRTLLM-5530][BREAKING CHANGE]: enhance the llm args pytorch config part 1(cuda_graph_config) (#5014 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-06-30 11:05:40 +08:00
Bo Li	6000380a0c	perf: Avoid reswizzle_sf after allgather. (#5504 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-06-29 21:25:50 +08:00
Talor Abramovich	70e34a3291	[TRTLLM-5831][feat] Add LoRA support for pytorch backend in trtllm-serve (#5376 ) Signed-off-by: Talor Abramovich <talora@nvidia.com>	2025-06-29 12:46:30 +00:00
amirkl94	de9779900c	feat: Add support for YARN in NemotronNAS models (#4906 ) Signed-off-by: Amir Klein <203507526+amirkl94@users.noreply.github.com>	2025-06-29 09:45:49 +03:00
Lucas Liebenwein	619709fc33	[AutoDeploy] merge feat/ad-2025-06-13 (#5556 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-06-29 03:52:14 +08:00
Li Min	6021a439ab	Make moe permute and final as custom op (#5412 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-06-27 15:48:33 -07:00
Darragh Hanley	5437075def	ReDrafter support for Qwen (#4875 ) Signed-off-by: darraghdog <darragh.hanley@gmail.com> Signed-off-by: Darragh Hanley <darragh.hanley@gmail.com> Co-authored-by: rakib-hasan <rhasan@nvidia.com>	2025-06-28 02:33:10 +08:00
Robin Kobus	a8141a4513	refactor: Speculative decoding buffers part 2 (#5316 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-27 17:41:48 +02:00
Aurelien Chartier	833c0dea4a	[TRTLLM-6104] feat: add request_perf_metrics to LLMAPI (#5497 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-06-27 17:03:05 +02:00
wili	56cdfe5c6c	[TRTLLM-5000][feat] NGrams V2 (#4569 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-06-27 23:00:17 +08:00
peaceh-nv	cb58073ab7	Fix : fix build for sm120 (#5265 ) Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>	2025-06-27 20:42:47 +08:00
Daniel Cámpora	73b8a95049	feat: Use inference mode in update_requests to improve perf of TRTLLM Sampler (#5538 )	2025-06-27 18:40:53 +08:00
Daniel Stokes	83a1f60556	feat: Expose bias and FP8_MXFP4 MOE CUTLASS backend features to pytorch (#5410 ) Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>	2025-06-27 12:29:34 +08:00
Yuxian Qiu	dc36228f52	fix: Fix block scale fp8 support for deepseek v3 on Blackwell. (#5514 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-06-27 11:03:38 +08:00
Yibin Li	0f3bd7800e	[TRTLLM-4971]: Use safe deserialization in ParallelConfig (#4630 ) Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>	2025-06-27 09:58:41 +08:00
Frank	aa6e015ef8	Update trtllm-bench to support new Pytorch default. (#5491 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-06-26 17:05:43 -07:00
Venky	0083228d2a	fix: Mapping rank boundary check bug (#4935 ) Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>	2025-06-27 07:27:59 +08:00
jmydurant	8836990bde	[TRTLLM-3602][feat] support nvfp4 model and fp8 kv cache for MLA chunked prefill (Blackwell) (#5475 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-06-26 22:18:08 +08:00
Robin Kobus	8dfa31c71d	refactor: remove batch_manager::KvCacheConfig and use executor::KvCacheConfig instead (#5384 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-26 19:45:52 +08:00
Rashid Kaleem	3a1f4d4001	[feat] Add progress bar to benchmark (#5173 ) Signed-off-by: Rashid Kaleem <rkaleem@nvidia.com> Co-authored-by: Zhihan Jiang <68881590+nvzhihanj@users.noreply.github.com> Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com>	2025-06-26 18:39:45 +08:00
Kaiyu Xie	2eb6502b1d	feat: Add support for TRTLLM CustomDataset (#5511 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-06-26 18:27:37 +08:00
Bo Li	1bab9000a6	perf: Optimize swizzle_sf, unswizzle_sf, reswizzle_sf (#5318 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-06-26 14:03:56 +08:00
dongxuy04	490d2e5819	feat: large-scale EP(part 8: Online EP load balancer integration for PCIe fp8) (#5226 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-06-25 22:25:13 -07:00
amitz-nv	e0bb123ae7	[TRTLLM-5921][feat] Prevent serialization of entire LoRA adapters in each request (#5080 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-06-26 08:15:06 +03:00
Yukun He	9ee33605bb	[TRTLLM-6019] feat: Remove cutlass min latency code from AutoTuner. (#5394 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-06-26 13:12:03 +08:00
Netanel Haber	6aef14943c	Revert "feature: unify new_tokens format sample state to trtllm samper new_tokens format (#4401 )" (#5474 ) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-06-25 20:56:04 -07:00
jmydurant	578dbc8d9a	feat: chunked prefill for MLA (Blackwell) (#4651 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-06-26 09:01:00 +08:00
Yukun He	3fc57543e2	[5356427] fix: Remove the seq_len of 4096 from FP8 block scale MoE tuning configs. (#5485 ) The seq_len of 4096 will cause some unknown CUDA illegal memory access issue if run with some other tests consecutively. Put a saturated upper bound for any sequence length larger than it.	2025-06-26 08:38:35 +08:00
Xianjie Qiao	1e4fa13d33	Add sleep function for disagg gen-only benchmarking (#5398 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-06-26 07:32:16 +08:00
QI JUN	3a2c4ca77b	chore: split _build_model method for TorchLlm and TrtLlm (#5418 ) Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-06-26 04:32:46 +08:00
Mike Iovine	5bc8c894f7	[chore] Disable block reuse when draft model speculation is being used (#5448 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-06-26 03:51:20 +08:00
Daniel Cámpora	205c97a4ae	[TRTLLM-5974][feat] Support disaggregated serving in TRTLLM Sampler (#5328 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> Signed-off-by: Daniel Cámpora <961215+dcampora@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-06-25 17:41:36 +02:00
Kaiyu Xie	c5ae3272b9	feat: Make benchmark_serving part of the library (#5428 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-06-25 23:13:56 +08:00
HuiGao-NV	b3a4c1f404	feat: Remove not used padding_idx in models (#5385 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-06-25 17:19:59 +08:00
Yiqing Yan	f3cfe86dd1	chore: bump version to 1.0.0rc1 (#5460 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-06-25 16:21:34 +08:00
Enwei Zhu	fc7a81ceb0	test: Add LLGuidance test and refine guided decoding (#5348 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-25 14:12:56 +08:00
Enwei Zhu	76da7fed86	fix (NvBug 5354925): Fix static EPLB (#5411 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-25 13:14:40 +08:00
Shunkangz	d5354897c0	feat: Dynamically remove servers in PD (#5270 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-06-25 09:50:04 +08:00
Lucas Liebenwein	5cffb7e0ec	[AutoDeploy] Merge feat/ad_2025_06_13 feature branch (#5454 ) Signed-off-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com> Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Co-authored-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com> Co-authored-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-06-25 09:30:13 +08:00
bhsueh_NV	73ba4fc320	fix: fix bug of qwen3 + eagle3 + finalize_moe_fusion (#5369 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-06-25 09:20:23 +08:00
dongxuy04	699520082b	Add MTP support for Online EPLB (#5213 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-06-25 07:58:13 +08:00
QI JUN	d93a5e04b5	Chore: remove unused variables (#5314 ) Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-06-24 22:27:32 +08:00
HuiGao-NV	35a92f6bab	Add debug hook to support dump tensor data and add new debug functions easily (#5182 ) Signed-off-by: Hui Gao	2025-06-24 17:45:28 +08:00
Luis Vega	d26040e5d9	chore: delete mamba hybrid, since it is now called NemotronH (#5409 ) Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>	2025-06-24 16:27:31 +08:00
Robin Kobus	e2a8cbc80b	refactor: manage cache indirection in decoder state (#5315 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-24 09:15:59 +02:00
HuiGao-NV	e16c1bef6e	[fix] Add 1 and draft_token_num to seq_len when overlap scheduling is enabled during memory estimation (#5343 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-06-24 11:43:43 +08:00
Netanel Haber	58a8a8fd37	feature: unify new_tokens format sample state to trtllm sampler new_tokens format (#4401 ) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-06-23 10:38:37 -07:00
dongxuy04	4f0f17ac8a	feat: Misc Opt for large scale EP (#5374 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-06-20 13:11:31 +08:00
Fanrong Li	5d4ab47d5b	fix: refactor and fix mtp vanilla (#4762 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-20 05:23:39 +08:00
Yan Chunwei	9bd42ecf9b	[TRTLLM-5208][BREAKING CHANGE] chore: make pytorch LLM the default (#5312 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-20 03:01:10 +08:00
Kaiyu Xie	7246fd75d1	feat: Support stream_interval (#5284 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-06-19 21:57:10 +08:00
Fanrong Li	c7af650d5a	Fix: fix the deterministic issue in the MTP Eagle path (#5285 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-19 18:08:40 +08:00
Frank	68687a9f56	[WAR][nvbug/5321947] Add an async sleep to unblock event loop. (#5342 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-06-19 17:25:18 +08:00
hlu1	b558232ce1	Refactor CutlassFusedMoE (#5344 ) Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>	2025-06-19 00:04:07 -07:00
amitz-nv	1753202b61	[TRTLLM-5825][fix] Fix torch LoRA TP (#5338 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-06-19 09:12:00 +03:00
Yiqing Yan	dedce8ab0e	chore: bump version to 1.0.0rc0 (#5326 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-06-19 12:02:28 +08:00
nv-guomingz	6a388b105a	chore: remove torch_compile prefix for TorchCompileConfig field members (#5261 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-06-19 09:21:51 +08:00
Zongfei Jing	2b23cd56ce	[feat] Fusion finalize and allreduce for qwenmoe model (#5223 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> Co-authored-by: Kefeng-Duan <176893526+Kefeng-Duan@users.noreply.github.com>	2025-06-19 08:03:58 +08:00
Yan Chunwei	3946e798db	fix[nvbug5298640]: trtllm-llmapi-launch multiple LLM instances (#4727 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-19 06:13:53 +08:00
jellysnack	0623ffe3bc	feat: Add LLGuidance Support for PyTorch Backend (#5214 ) Signed-off-by: jellysnack <oleg.jellysnack@gmail.com> Signed-off-by: jellysnack <158609015+jellysnack@users.noreply.github.com> Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-18 19:33:34 +08:00
Zhanrui Sun	516bd4dc05	chore: bump version to 0.21.0rc3 (#5309 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-06-18 15:59:53 +08:00

... 12 13 14 15 16 ...

1988 Commits