TensorRT-LLMs

mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-02-14 23:14:02 +08:00

Author	SHA1	Message	Date
Pengbo Wang @ NVIDIA	ffc976ceaf	[https://nvbugs/5445466 ][fix] fix deepseek r1 hang by not enabling mnnvl by default (#6860 ) Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com> Co-authored-by: Tao Li @ NVIDIA <tali@nvidia.com>	2025-08-14 22:36:56 +08:00
Shi Xiaowei	1095dfd03c	[None][fix] BREAKING CHANGE: Mismatch between docs and actual commands (#6323 )	2025-08-14 03:48:57 -04:00
Yan Chunwei	0132c1db84	[https://nvbugs/5427043 ][fix] request length exceeds max_num_tokens (#6821 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-08-14 13:31:12 +08:00
Bo Deng	d8acca495b	[TRTLLM-6675][infra] Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6623 (#6735 ) Signed-off-by: Bo Deng <deemod@nvidia.com>	2025-08-14 04:36:38 +00:00
jmydurant	4200fa46d1	[None][feat] Add support for Hopper MLA chunked prefill (#6655 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-08-14 10:39:26 +08:00
Izzy Putterman	ef53de8eef	[None][feat] Add test for speculative rejection sampler (2-model) (#6542 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-08-13 22:09:35 -04:00
Tin-Yin Lai	6c52bb07ff	[https://nvbugs/5302040 ][feat] Add whisper support (Bert Attention on SM100 and GPTAttention for cross attention on SM100) (#5527 ) Signed-off-by: tinyinl <tinyinl@nvidia.com>	2025-08-13 11:19:13 -07:00
danielafrimi	bda42f8c3a	[None][feat] Support running heterogeneous model execution for Nemotron-H (#6866 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-08-13 19:51:19 +03:00
Anthony Chang	2198587b35	[https://nvbugs/5378031 ] [feat] Hopper W4A8 MoE supports ModelOpt ckpt for PyT backend (#6200 ) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>	2025-08-13 21:24:40 +08:00
Yukun He	bc5f766e0e	[TRTLLM-4501][feat] AutoTuner tuning config refactor and valid tactic generalization. (#6545 ) * Generalize the definition of tactics so that users can implement more customizable tactic types, making the configurations clearer for each kernel run. * Allow the user not to specify the `gen_tuning_buckets` or the `map_to_tuning_buckets` function. * Other code refactoring. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-13 16:25:22 +08:00
Void	1d80df0955	[None][feat] DeepEP LL combine FP4 (#6822 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-08-13 04:20:21 -04:00
Mike Iovine	f68e03e646	[https://nvbugs/5452167 ][fix] Fix ngram padding issue (#6837 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-13 11:23:16 +08:00
Yechan Kim	12102e2d48	[TRTLLM-6772][feat] Multimodal benchmark_serving support (#6622 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-12 19:34:02 -07:00
Fanrong Li	1bbc0e323b	[None][fix] Pre-allocate workspaces for DeepGEMM MoE to avoid frequent cudaFree/cudaMalloc (#6811 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-08-13 10:27:57 +08:00
rakib-hasan	2923eb88a1	[None][fix] Refactoring input prep to allow out-of-tree models (#6497 ) Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-08-12 20:29:10 -04:00
dongxuy04	bd9a6dd9ab	[TRTLLM-7008][fix] fix wideEP weights loading and args (#6789 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-08-12 19:14:20 -04:00
Robin Kobus	45c7518032	[None][refactor] Simplify decoder state initialization (#6559 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-12 21:44:41 +02:00
Robin Kobus	dd11e08d26	[#6187 ][feat] add LayerNorm module (#6625 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-12 21:43:30 +02:00
nvchenghaoz	81f0ded1c4	[None][feat] Add GPT OSS support for AutoDeploy (#6641 ) Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com>	2025-08-12 14:03:22 -04:00
Jhao-Ting Chen	a060e12041	[https://nvbugs/5438869 ][fix] Set nvfp4 expert w1 w3 weight scale to the same value if they're not (#6656 ) Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>	2025-08-12 20:47:10 +08:00
Shunkangz	ab0d768acf	[None][fix] Fix attention dp log (#6570 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-08-12 04:53:09 -04:00
Liao Lanyu	f7c13a4aa7	[TRTLLM-6906][chore] Using pybind to bind functions in thop/attentionOp (#6745 ) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>	2025-08-12 16:45:16 +08:00
Sergey Klevtsov	27fc35175e	[None][feat] CUTLASS MoE FC2+Finalize fusion (#3294 ) Signed-off-by: Sergey Klevtsov <sklevtsov@nvidia.com>	2025-08-12 15:56:48 +08:00
Fridah-nv	0dc4b4e699	[#4403 ][autodeploy] Refactor: Move more transformations to new inf optimizer, Add quantization_source to factory interface (#6760 ) Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>	2025-08-11 22:02:46 -07:00
Enwei Zhu	7c686ba8de	[TRTLLM-2285][feat] Enable guided decoding with CUDA graph padding and draft model chunked prefill (#6774 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-12 09:30:06 +08:00
Ziyi Xiong	b4fcd5f592	[https://nvbugs/5441438 ][fix] Set correct draft length for the cuda graph dummy request (#6701 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-08-12 09:28:47 +08:00
Jinyang Yuan	ead89a0e40	[None][perf] Improve the performance of online EPLB on Hopper by better overlapping (#6624 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-08-12 09:25:13 +08:00
Chang Liu	be9dd4713c	[https://nvbugs/5385987 ][fix] Fix Qwen2 quantization issue by pinning transformers version (#6673 ) Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com>	2025-08-11 17:16:49 -07:00
rakib-hasan	7ab8112450	[None][fix] Refactoring to avoid circular import when importing torch models (#6720 ) Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-08-11 18:00:42 -04:00
bhsueh_NV	83dbc6c75d	[TRTLLM-5532][feat] store the block of context request into kv cache (#6683 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-08-11 16:14:52 +08:00
Tracin	49bcaa4e95	Add gpt-oss GSM8K test. (#6732 ) Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>	2025-08-10 22:45:43 -04:00
Zero Zeng	4b4b91ab51	[None][feat] improve dataloading for benchmark_dataset by using batch… (#6548 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-08-11 09:50:41 +08:00
Yechan Kim	60073a7ad9	[None][feat] Support SharedTensor on MultimodalParams (#6254 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-10 17:48:24 -07:00
shaharmor98	b6baa9ed9b	[TRTLLM-6823][doc] Add checkpoint refactor docs (#6592 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-08-10 19:47:39 -04:00
shaharmor98	14b36e07d7	[TRTLLM-6174][feat] Enable FP32 mamba ssm cache (#6574 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-08-10 16:27:51 -04:00
Gal Hubara-Agam	3c5aec19c2	[#5048 ][enhance] AutoDeploy: Optimize prepare_inputs (#6634 ) Optimize prepare_inputs routine in AutoDeploy, as part of the effort to reduce the performance gap compared to the default backend. This PR includes two major fixes, and some other minor tweaks: 1. Avoid back and forth data copies 2. Optimize position ids update by separating the implementation for generation mode and context mode. Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>	2025-08-10 13:55:04 +03:00
Ziyi Xiong	de472828b9	[TRTLLM-6637][feat] Resolve KV cache divergence issue (#6628 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-08-09 23:15:04 +08:00
Yilin Fan	d643aef73c	[Perf] Improve Llama4 performance for small max_seqlen cases (#6306 ) Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>	2025-08-09 02:58:31 -04:00
Ye Zhang	bcf5ec0c9a	[None][feat] Core Metrics Implementation (#5785 ) Signed-off-by: Ye Zhang <zhysishu@gmail.com> Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-08-09 02:48:53 -04:00
Yibin Li	97787883c3	[TRTLLM-6420][feat] add support for Eclairv2 model - cherry-pick changes and minor fix (#6493 ) Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>	2025-08-08 21:40:48 -04:00
dongfengy	d06675071e	[None][fix] WAR GPT OSS on H20 with Triton MOE (#6721 ) Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>	2025-08-08 19:47:09 -04:00
Mike Iovine	90145cf557	[None][feat] Optimize CUDA graph memory usage for spec decode cases (#6718 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-08 13:56:53 -04:00
Wanli Jiang	d45236b253	[TRTLLM-6308][feat] Support Aggregate mode for phi4-mm (#6184 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-08-08 20:09:26 +08:00
Stefan Niebler	b8f036f264	[TRTLLM-6650][fix] Enhance CUDA graph + Beam search to correctly handle padding (#6665 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-08-08 14:00:33 +02:00
Liao Lanyu	32ad7f3c12	[None][fix] Remove lock related typo in py_executor (#6653 ) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com>	2025-08-08 17:48:57 +08:00
JunyiXu-nv	5f45227a93	[https://nvbugs/5437106 ][fix] Fix llama4 scout TRTLLM attn_backend (#6690 ) Signed-off-by: Junyi Xu <junyix@nvidia.com>	2025-08-08 17:48:23 +08:00
Yuxian Qiu	9ff4e75f14	[None][refactor] Combine resmooth_to_fp8_e8m0 and transform_sf_into_required_layout (#6654 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-08-08 17:11:41 +08:00
Li Min	d913955952	[TRTLLM-6898][feat] make fused_moe_cute_dsl work on blackwell (#6616 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-08-08 15:03:48 +08:00
2ez4bz	064eb7a70f	[TRTLLM-5252][fix] Propagate mapping to intermediate layers (#6611 ) This commit propagates the mapping to intermediate layers to enable tensor parallelism (amongst other things) in them. It also fixes issues with a unit test for TP for pixtral, and adds it to a test list. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-08-08 01:50:36 -04:00
Enwei Zhu	aee828d98a	[TRTLLM-6854][feat] Enable guided decoding with disagg serving (#6704 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-08 12:10:36 +08:00
zhanghaotong	1cf669496a	[None][fix] Fix unnecessary GPU synchronization in torch sampler caused by incorrect tensor reference (#6626 ) Signed-off-by: 皓聪 <zhanghaotong.zht@alibaba-inc.com> Co-authored-by: 皓聪 <zhanghaotong.zht@alibaba-inc.com>	2025-08-07 23:44:47 -04:00
NVJiangShao	2f2f5cc72c	[TRTLLM-6744][feat] Remove input_sf swizzle for module WideEPMoE (#6231 ) Signed-off-by: Jiang Shao <91270701+StudyingShao@users.noreply.github.com>	2025-08-08 11:13:42 +08:00
Daniel Cámpora	efca359b66	[TRTLLM-6785][feat] BREAKING CHANGE Enable TRTLLM sampler by default (#6216 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-08-07 22:19:37 -04:00
Iman Tabrizian	82276167e6	[None][feat] Add NCCL Symmetric Integration for All Reduce (#4500 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-08-07 17:28:14 -07:00
Haohang Huang	980929e1a9	[https://nvbugs/5410687 ][fix] Hopper w4a8 groupwise MoE interleave (#6708 ) Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-08-07 15:30:16 -07:00
Yuan Tong	db8dc97b7b	[None][fix] Migrate to new cuda binding package name (#6700 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-08-07 16:29:55 -04:00
Mike Iovine	e968f98b43	[None][feat] Clean up ngram auto mode, add max_concurrency to configs (#6676 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-08-07 12:51:47 -04:00
Emma Qiao	3c44b44e45	[None][infra] Fix guardwords (#6711 ) Signed-off-by: qqiao <qqiao@nvidia.com>	2025-08-07 21:06:47 +08:00
pcastonguay	453a06e6ab	[TRTLLM-6881][feat] Include attention dp rank info with KV cache events (#6563 ) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>	2025-08-07 14:17:07 +02:00
Enwei Zhu	1b9781e8e7	[TRTLLM-6409][feat] Enable guided decoding with speculative decoding (part 1: two-model engine) (#6300 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-08-07 05:53:48 -04:00
hlu1	8207d5fd39	[None] [feat] Add model gpt-oss (#6645 ) Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>	2025-08-07 03:04:18 -04:00
amitz-nv	85af62184b	[TRTLLM-6683][feat] Support LoRA reload CPU cache evicted adapter (#6510 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-08-07 09:05:36 +03:00
Yiqing Yan	5fa1914cab	[None][chore] Bump version to 1.1.0rc0 (#6651 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-07 13:39:49 +08:00
Izzy Putterman	7e0158b583	Qwen3: Fix eagle hidden states (#6199 ) Signed-off-by: Izzy Putterman <iputterman@nvidia.com>	2025-08-06 17:05:18 -04:00
Hanjun Cho	80f918cc22	[None][feat] Add Qwen3 MoE support to TensorRT backend (#6470 ) Signed-off-by: gkswns0531 <gkswns0531@gmail.com> Signed-off-by: hanjuncho <gkswns0531@gmail.com> Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>	2025-08-06 17:02:35 +08:00
Zongfei Jing	0ff8df95b7	[https://nvbugs/5433581 ][fix] DeepGEMM installation on SBSA (#6588 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-08-06 16:44:21 +08:00
Netanel Haber	83ee91e17b	[None][fix] Fix 6522 mpi.pkl5.intracomm.Request has wait not Wait (#6646 ) Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-08-06 14:18:09 +08:00
JunyiXu-nv	13e0214fe0	[TRTLLM-6263][feat] Enable fp8 SwiGLU to minimize host overhead (#6540 ) Signed-off-by: Junyi Xu <junyix@nvidia.com>	2025-08-06 10:42:19 +08:00
brb-nv	9a01934dbf	[None][feat] Switch to internal version of MMProjector in Gemma3 (#6572 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-08-05 21:48:23 -04:00
yunruis	3ff4f503ad	[None][opt] ADP schedule balance optimization (#6061 ) Signed-off-by: yunruis <205571022+yunruis@users.noreply.github.com>	2025-08-06 09:38:02 +08:00
Yechan Kim	c17f4984e2	[None][feat] Refactor Llava-Next (#6478 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-05 17:53:53 -07:00
Aurelien Chartier	6da95f29a9	[None][feat] Add support for fused gate_up_proj scales for FP8 blockwise (#6496 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-08-05 11:22:32 -07:00
Wanli Jiang	46df8712c8	[https://nvbugs/5355007 ][fix] Set `enable_chunked_context` as True by default in trtllm bench (#6582 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-08-05 11:11:36 -07:00
ixlmar	1ebceb790d	[TRTLLM-5508][feat] check input tokens + improve error handling (#5170 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-08-05 18:27:43 +01:00
liji-nv	dcbfa7e509	[https://nvbugs/5252313 ][fix] Fix torch compile + MTP (#6554 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-08-05 10:31:29 -04:00
Venky	61da2daeb4	[TRTLLM-6761][refactor] Replace LogitBiasLogitsProcessor with embedding bias tensor system (#6464 ) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>	2025-08-05 07:14:24 -07:00
Pengbo Wang @ NVIDIA	c289880afb	[None][fix] fix kimi k2 serving and add test for Kimi-K2 (#6589 ) Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>	2025-08-05 18:05:33 +08:00
amitz-nv	dc84695520	[TRTLLM-6826][feat] Allow sending more than 2GiB through MPI by using mpi4py.util.pkl5 (#6522 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-08-05 11:28:26 +03:00
danielafrimi	ed801ff74b	[None][fix] Remove expand configuration from mamba2 mixer (#6521 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-08-05 04:18:25 -04:00
Haohang Huang	c9eebcb454	[TRTLLM-6674][feat] (Breaking Change) Hopper SWA non-cyclic kernels + KV reuse + Spec Dec (#6379 ) Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com> Signed-off-by: symphonylyh <31998628+symphonylyh@users.noreply.github.com>	2025-08-05 07:47:41 +00:00
kris1025	6a3a921284	[TRTLLM-6685][feat] Add speculative metrics for trt llm bench (#6476 ) Signed-off-by: linquanh <linquanh@nvidia.com>	2025-08-04 15:22:57 -07:00
Olya Kozlova	13cc1c4878	[TRTLLM-5271][feat] best_of/n for pytorch workflow (#5997 ) Signed-off-by: Olya Kozlova <okozlova@nvidia.com>	2025-08-04 14:08:06 +02:00
brb-nv	87e4e9f468	[None][chore] Add unit test for Gemma3 lora (#6560 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-08-04 04:56:57 -04:00
Yiqing Yan	3916dbd98b	[None][chore] Bump version to 1.0.0rc6 (#6597 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-08-04 04:39:15 -04:00
Pengyun Lin	a15e33351d	[None][fix] Revert commit `48ddc3d` & add test for disagg server with different max_num_tokens (#6259 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-08-04 15:09:51 +08:00
Yuan Tong	a2f271c8e0	[TRTLLM-4406][feat] LLM sleep & wakeup Part 1: virtual device memory (#5034 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-08-04 13:51:01 +08:00
Yechan Kim	ee6ab5be96	chore: add EXAONE4 accuracy test (#6397 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-08-04 10:14:16 +08:00
Jinyang Yuan	df90202b51	[fix] Fix DeepSeek w4a8 weight loading (#6498 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-08-04 10:12:06 +08:00
Chuang Zhu	542f552d0b	use cudaSetDevice to create context ,fix nvbug 5394497 (#6403 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-08-03 13:32:55 -04:00
Shunkangz	67a3fd858b	[None][feat] Add support of scheduling attention dp request (#6246 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Co-authored-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>	2025-08-01 20:38:01 -04:00
Richard Huo	31802de0b0	[None][fix] Serialize the window_size in the kv event (#6526 ) Signed-off-by: richardhuo-nv <rihuo@nvidia.com>	2025-08-01 15:25:18 -07:00
Lucas Liebenwein	5247df6ae2	[AutoDeploy] merge feat/ad-2025-07-22 (#6520 ) Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Gal Agam <ghubaraagam@cw-dfw-cs-001-login-01.cm.cluster> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: haoguo <67671475+h-guo18@users.noreply.github.com> Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Co-authored-by: Gal Agam <ghubaraagam@cw-dfw-h100-004-328-012.cm.cluster> Co-authored-by: h-guo18 <67671475+h-guo18@users.noreply.github.com> Co-authored-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>	2025-08-01 08:51:08 -07:00
brb-nv	7447d6ed85	[TRTLLM-6657][feat] Add LoRA support for Gemma3 (#6371 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-08-01 09:19:54 -04:00
liji-nv	1daa8c3232	[https://nvbugs/5340941 ][https://nvbugs/5375785 ] - fix: Wrap attentio… (#6355 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-08-01 07:38:06 -04:00
Yukun He	90856bf97d	[https://nvbugs/5419069 ][fix] Fix the mismatched layer name components. (#6417 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-01 16:32:39 +08:00
Zero Zeng	48768fd720	fix: Fix missing key (#6471 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-08-01 14:25:58 +08:00
Robin Kobus	d3c14682f0	refactor: Remove unused buffers and bindings from sampler (#6484 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-08-01 00:43:03 -04:00
Jaedeok Kim	fbee279909	fix: remove duplicate layer multiplication in KV cache size calculation (#6481 ) Signed-off-by: Jaedeok Kim <jaedeokk@nvidia.com>	2025-07-31 22:34:34 -04:00
Zongfei Jing	7bb0a78631	Deepseek R1 FP8 Support on Blackwell (#6486 ) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> Co-authored-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-08-01 10:26:28 +08:00
Venky	8c165fd27a	[TRTLLM-6611][feat] Add warnings and stricter validation to LoraManager adapter loading (#6453 ) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com>	2025-07-31 22:22:51 -04:00
Yukun He	00059de380	chore: Improve the AutoTuner log information. (#6368 ) * Change the fallback alert from DEBUG to WARNING level and only do it once. * Add debug information for profiling cache right after the warmup phase. * Change the level of exception message during tactic profiling from ERROR to WARNING level. All exception details are pushed to the DEBUG level. * Other trivial refinements and cleanups. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-08-01 09:19:52 +08:00
brb-nv	2eca0d5925	fix: Fix poor generation with FP8 Gemma3 1B checkpoint (#6499 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-31 17:18:23 -07:00
Simeng Liu	8cf3faa26a	[feat] Auto-enable ngram with concurrency <= 32. (#6232 ) Signed-off-by: Simeng Liu <simengl@nvidia.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> Signed-off-by: Mike Iovine <mike.iovine7@gmail.com> Co-authored-by: Mike Iovine <miovine@nvidia.com> Co-authored-by: Mike Iovine <mike.iovine7@gmail.com>	2025-07-31 18:45:51 -04:00
Ziyi Xiong	8062e0fe7c	[TRTLLM-6392][feat] Support turning on/off spec decoding dynamically (#6363 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-31 15:31:39 -04:00
shaharmor98	0c42f54a39	Bugfix/fix nemotron nas lora support (#6380 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-07-31 13:39:35 -04:00
amitz-nv	1ee7a08d2b	[5830][feat] Improve LoRA cache memory control (#6220 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-07-31 09:26:38 +03:00
dongjiyingdjy	17e0d0fb1a	fix: fix illeagel memory access (#6437 ) Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>	2025-07-31 10:01:34 +08:00
Enwei Zhu	4b299cb77e	feat: Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 (#6408 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-31 09:53:52 +08:00
Vadim Gimpelson	25cd4f215e	[PERF] Move calculation Qwen2-VL's rotary_cos_sin to LLM worker process (#6004 ) Signed-off-by: Vadim Gimpelson <vadim.gimpelson@centml.ai>	2025-07-31 09:35:24 +09:00
shaharmor98	f9cf683e39	add propagation of trust_remote_code to OpenAIServer (#6446 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-07-30 15:25:41 -04:00
Wanli Jiang	9632dba02e	feat: TRTLLM-6450 update long rope for phi3.5/phi4-mini/phi4-mm (#6353 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-30 09:20:16 -07:00
NVShreyas	e67f4da9b5	[Perf]: Add residual, norm for nemotron_nas models (#6455 ) Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>	2025-07-30 09:10:38 -07:00
Chang Liu	b4065d8ca6	[TRTLLM-6654][feat] Add support for external multimodal embeddings (#6263 ) Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>	2025-07-30 10:00:15 -04:00
pcastonguay	e7ae5e2824	feat: Add support for disaggregation with pp with pytorch backend (#6369 ) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> Signed-off-by: raayandhar <rdhar@nvidia.com> Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> Signed-off-by: pcastonguay <55748270+pcastonguay@users.noreply.github.com> Co-authored-by: raayandhar <rdhar@nvidia.com> Co-authored-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2025-07-30 09:42:13 -04:00
tomeras91	a2514d93fc	[nvbug 5380101][fix] Fix nemotronNAS loading for TP>1 (#6447 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-07-30 07:22:32 -04:00
QI JUN	2fe9cc0889	chore: remove draft_model_engine from init parameter list of PyExecutor (#6325 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-30 03:31:49 -04:00
QI JUN	1f39a11af0	chore: clean code of PyExecutor (#6445 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-30 02:11:43 -04:00
2ez4bz	d6eed1b624	[fix] Switch placement of image placeholder for mistral 3.1 (#6435 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-30 14:10:36 +08:00
Jinyang Yuan	a427f5bece	[fix] Fix wide EP when using DeepEP with online EPLB (#6429 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-07-30 00:13:18 -04:00
Zheng Duan	c9ed1ab436	[TRTLLM-6549] chore: record delay introduced by disaggregated serving in kv cache measure (#6135 ) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>	2025-07-30 10:39:40 +08:00
peaceh-nv	5b420ad267	Rename layer to comply with deepseek (#6393 ) Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>	2025-07-30 10:00:48 +08:00
Yechan Kim	d6eb8e2366	fix: support mixture of text & multimodal prompts (#6345 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-30 08:52:31 +08:00
Yunfan Fan	1a8e28d295	[FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine (#6344 ) Signed-off-by: fanyunfan <2569548856@qq.com> Co-authored-by: fanyunfan <2569658856@qq.com>	2025-07-30 07:13:44 +08:00
Yan Chunwei	ad662ddcdd	chore: disallow arbitrary in llm_args.Configs (#6367 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-29 16:16:52 -04:00
Michal Guzek	7efe3cb0cd	[fix] Add detokenization-based stop word logic to LLM API (#5948 ) Signed-off-by: moraxu <mguzek@nvidia.com> Signed-off-by: Michal Guzek <mguzek@nvidia.com>	2025-07-29 10:16:59 -07:00
Yukun He	0eee2e2850	[5385981] fix: Update the usage of VisionAttention init API. (#6413 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-07-29 16:41:48 +08:00
QI JUN	13e24ab1cb	chore: remove unused code in PyExecutor (#6351 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-29 16:24:26 +08:00
Frank	d2a04abb95	[fix] Fixes to parameter usage and low latency configuration. (#6343 )	2025-07-29 01:36:13 -04:00
nv-guomingz	49044733e1	chore: delete useless gitkeep files. (#6400 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-28 11:38:30 -04:00
QI JUN	4efc6496b7	chore: add _prepare_and_schedule_batch function in PyExecutor (#6365 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-28 05:50:27 -04:00
Yan Chunwei	45d441e60c	[TRTLLM-5061] chore: add status tags to LLM API reference (#5707 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-28 15:57:07 +08:00
Zero Zeng	c9b8b6180f	Add Acceptance Rate calculation to benchmark_serving (#6240 ) Signed-off-by: Zero Zeng <38289304+zerollzeng@users.noreply.github.com>	2025-07-28 14:00:58 +08:00
Jinyang Yuan	97f7e12588	[fix] Fix perf regression caused by MoE autotuner when using DeepEPLowLatency (#6288 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-07-28 01:37:11 -04:00
Chang Liu	dc757799e1	[nvbugs/5401156][fix] Avoid import all models when import trtllm._common (#6266 )	2025-07-27 23:29:21 -04:00
Void	f172face98	DeepEP LL dispatch FP4 (#6296 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-07-28 11:25:42 +08:00
Yukun He	93a0fd0a23	[TRTLLM-6445] feat: Enable AllReduce-associated fusion patterns in Llama3/4. (#6205 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-07-28 09:36:26 +08:00
YueWeng	2dd3186727	fix: remove cudaStreamSynchronize when using relaxed acceptance (#5262 ) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>	2025-07-28 09:18:41 +08:00
Ziyi Xiong	d853811190	[https://nvbugs/5402719 ][fix]: Add cuda graph dummy requests to the spec_resource_manager (#6258 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-26 20:32:39 -04:00
Michal Guzek	08d57123f9	[nvbug/5374773] chore: Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache (#5974 ) Signed-off-by: moraxu <mguzek@nvidia.com>	2025-07-25 18:10:40 -04:00
ameynaik-hub	1e5e71aa42	Mtp optimizations round1 (#5689 ) Signed-off-by: Amey Naik <212485788+ameynaik-hub@users.noreply.github.com> Co-authored-by: Kefeng-Duan <176893526+Kefeng-Duan@users.noreply.github.com>	2025-07-25 13:48:27 -04:00
nv-guomingz	b8d4cb8beb	feat: Support JSON Schema in OpenAI-Compatible API (#6321 ) Signed-off-by: noiji <52301388+noiji@users.noreply.github.com>	2025-07-25 12:55:56 -04:00
xiaoqi	a0aecf0476	[feat]: support logit_bias (#5354 ) Signed-off-by: xq25478 <xq25478@qq.com> Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> Signed-off-by: hexiao.xq <hexiao.xq@antgroup.com> Co-authored-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> Co-authored-by: hexiao.xq <hexiao.xq@antgroup.com> Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-25 09:37:41 +00:00
liji-nv	e07fff4f78	[https://nvbugs/5340941 ] - fix: Correct custom ops used by Qwen3 Moe … (#6285 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-25 14:49:45 +08:00
Mike Iovine	0f2f11f90b	[TRTLLM-6453][feat] Support chunked prefill on spec decode 2 model (#6104 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-24 21:50:11 -04:00
Linda	9a99e6d6d7	fix: integration tests with nanobind (#6326 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-07-25 09:23:20 +08:00
Shiyu Li	375f74ecb2	[fix][nvbugs/5399355] Fix Lamport buffer clear issue for MNNVL TwoShot Allreduce and add FP16 support. (#6237 ) Signed-off-by: Shiyu Li <shili@nvidia.com>	2025-07-25 08:01:40 +08:00
Frank	f8f5ba65fc	[fix] Update to remove popping of KV cache and other args. (#6310 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-24 15:54:33 -04:00
Stefan Niebler	0df758ec9f	[TRTLLM-6650][feat] Enhance beam search support with CUDA graph integration (#6217 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-24 18:04:41 +02:00
bhsueh_NV	7b6aadc800	[Fix][nvbug 5401163][nvbug 5404726][Qwen3] Fix bug of MoE on tp > 1 with trtllm moe backend (#6235 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-07-24 21:47:37 +08:00
liji-nv	14d94a3856	feat: Add non UB AR + Residual + Norm + Quant fusion (#6320 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-24 05:51:43 -04:00
Lizhi Zhou	a63a1ac7f9	[TRTLLM-6444] Add some UCX trouble shooting docs and print UCX related logs (#6085 ) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>	2025-07-24 16:21:01 +08:00
QI JUN	428e34080f	chore: remove unused variables in pyexecutor (#6280 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-24 13:16:15 +08:00
Stefan Niebler	2486eb778e	[TRTLLM-6651][feat] Enable Overlap scheduler + Beam Search in TRTLLM Sampler (#6223 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-23 12:30:50 +02:00
YueWeng	ed62a06eef	[nvbug/5322354] fix PD + MTP + overlap scheduler accuracy issue (#6136 ) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com>	2025-07-23 14:53:37 +08:00
QI JUN	a8253b942f	chore: remove duplicate should_stop_processing check (#6242 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-23 14:11:23 +08:00
Yechan Kim	83c3ed128b	chore: set default device to cpu on Multimodal models (#5994 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-22 21:45:31 -07:00
Erin	5636c67388	fix: nvbug_5398806 (#6239 )	2025-07-23 11:45:11 +08:00
Venky	9538c8d0e5	Add basic Nemo Ckpt Lora Loading in pytorch flow (#6019 )	2025-07-22 19:42:45 -07:00
wili	8ecdeee300	[refactor] Simplification of Speculative decoding configs - Part 2 (#5936 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-07-23 09:20:27 +08:00
Lucas Liebenwein	41fb8aa8b1	[AutoDeploy] merge feat/ad-2025-07-07 (#6196 ) Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com> Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com> Co-authored-by: Neta Zmora <nzmora@nvidia.com> Co-authored-by: nvchenghaoz <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Co-authored-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com>	2025-07-23 05:11:04 +08:00
2ez4bz	ab7434ac62	[feat] Enable TP and batching for PixtralVisionModel / Mistral3VLM (#6152 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-22 11:06:41 -07:00
John Calderon	b7c8a672da	[Issue 6193] Fix gemma3vl weight loader (#6233 ) Signed-off-by: John Calderon <johncalesp@gmail.com>	2025-07-22 10:32:18 -07:00
danielafrimi	ff9963978a	Add register_fake for finegrained_mixed_dtype_gemm torch_op (#6255 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-07-22 16:59:55 +03:00
Yiqing Yan	3e18ee5fe1	chore: bump version to 1.0.0rc5 (#6252 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-22 16:24:28 +08:00
Pengyun Lin	48ddc3d4b9	[fix]: Revert commit `388b491` (#6143 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-22 12:48:00 +08:00
Yi Zhang	eb7d0f84b5	[nvbugs/5368410][fix] Disable moe allreduce for multi node (#5918 ) Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>	2025-07-22 12:48:00 +08:00
Fanrong Li	c66941036f	fix: fix index out of bounds error in spec decoding (#5954 )	2025-07-22 12:48:00 +08:00
Shunkangz	ee45e0c63f	feat: Refactor the fetching request logic (#5786 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-07-22 09:16:28 +08:00
Chang Liu	7381f1dba7	[TRTLLM-5059][feat] Add KV cache reuse support for multimodal models (#5444 ) Only supports qwen in this PR	2025-07-21 16:11:58 -07:00
Ziyi Xiong	d7f0b0ab68	[fix] Correct the returned value of has_spec_drafter (#6178 ) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-21 11:38:59 -04:00
Pengyun Lin	9832bef07d	[BREAKING CHANGE]: change default backend to PyTorch in trtllm-serve (#5717 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-21 21:09:43 +08:00
liji-nv	3e0fb60e50	[TRTLLM-4279] feat: Multistream initial support for torch compile flow (#5847 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-21 19:10:22 +08:00
Linda	3efad2e58c	feat: nanobind bindings (#6185 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-07-21 08:56:57 +01:00
Yuening Li	e8c068b4b1	[TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow (#5850 ) Signed-off-by: Yuening Li <62227368+yueningl@users.noreply.github.com> Co-authored-by: Yuening Li <62227368+yueningl@users.noreply.github.com>	2025-07-21 15:17:35 +08:00
Jinyang Yuan	88076eecd0	[fix] Fix can_use_alltoall in fused_moe_wide_ep.py (#6173 ) Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>	2025-07-21 10:53:07 +08:00
brb-nv	ca9bc5727e	fix: Flush stale `PlanParams` with custom attention mask (#6163 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-21 09:55:09 +08:00
brb-nv	a433ebad2b	enh: Lift expectation of single image per sample in Gemma3 VLM (#6195 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-21 08:43:07 +08:00
danielafrimi	5300a99bd8	W4A8 GEMM (#6005 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-07-20 17:34:57 +03:00
amitz-nv	98428f330e	[TRTLLM-5826][feat] Support pytorch LoRA adapter eviction (#5616 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-07-20 08:00:14 +03:00
Void	118307c224	DeepEP LL support variable hidden size and tokens num (#6141 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-07-20 09:32:41 +08:00
Pengyun Lin	69e9f6d489	[fix]: Skip prompt length checking for generation only requests (#6146 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-19 21:26:37 +08:00
Ziyi Xiong	66030ef815	[TRTLLM-6452][feat]: Two-model engine KV cache reuse support (#6133 ) Signed-off-by: ziyixiong-nv <fxiong@nvidia.com> Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>	2025-07-19 13:17:15 +08:00
Rashid Kaleem	152e2df43b	[Disaggregated] Add retry knobs and handling (#5808 ) Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com> Signed-off-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com>	2025-07-19 07:27:59 +08:00
John Calderon	fc8b29c4ff	[Issue 5927][fix] Avoid memory calls during broadcast for single GPU (#6010 ) Signed-off-by: John Calderon <johncalesp@gmail.com>	2025-07-18 14:21:03 -07:00
Netanel Haber	d9a3530048	[nvbug/5393888][nvbug/5393042] Always use `py_seq_slot` (#6147 ) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-07-18 22:45:16 +03:00
Stefan Niebler	6d7874a467	[nvbugs/5369799] fix: Update disaggregation handling in sampler (#5762 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-19 01:40:46 +08:00
xiaoqi	28858c8711	feat(eagle3):support qwen3 dense model (#5879 ) Signed-off-by: xq25478 <xq25478@qq.com>	2025-07-19 01:24:32 +08:00
Bo Li	07e8813984	feat: Remove padding in attention DP. (#6064 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-07-18 23:30:34 +08:00
Stefan Niebler	fd6ce7f20e	[ci] Speedup beam search unit tests with fixtures for LLM (#5843 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-18 22:54:49 +08:00
Erin	9522cde464	fix: NVBug 5385576 py_batch_idx issue (#6153 ) Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>	2025-07-18 22:36:43 +08:00
Robin Kobus	ec2b953e7e	refactor: Enhanced handling of decoder requests and logits within the batch manager (#6055 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-18 12:12:08 +02:00
Aurelien Chartier	812243bdd6	feat: add support for Modelopt fp8_pb_wo quantization scheme (#6106 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-07-18 10:35:12 +08:00
Zhenhuan Chen	992b273045	[https://nvbugs/5387375 ] fix(scaffolding): fix scaffolding aime test in test_e2e (#6140 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-18 10:34:37 +08:00
yifeizhang-c	0155e7a3a1	[TRTLLM-6368] Update deepep dispatch API (#6037 ) Signed-off-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>	2025-07-18 10:13:31 +08:00
Iman Tabrizian	b75e53ab69	Revert "feat: nanobind bindings (#5961 )" (#6160 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-18 10:12:54 +08:00
qixiang-99	2c90203c36	Refactor KVCacheManager: Simplify token availability calculation and … (#6134 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-07-17 13:33:33 -07:00
Frank	161490f039	[fix] Fixes KV Cache overrides in trtllm-bench (#6103 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-18 03:44:44 +08:00
2ez4bz	8480c120b1	[fix] Fix Mistral3VLM weight-loading & enable in pre-merge (#6105 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-17 11:04:17 -07:00
Iman Tabrizian	10dbf4f0f4	[fix] Remove duplicated KVCache transmission check (#6022 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-17 12:02:19 -04:00
Linda	5bff317abf	feat: nanobind bindings (#5961 ) Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>	2025-07-17 22:42:52 +08:00
Ziyi Xiong	58d22a72f1	[TRTLLM-6352][feat] Migrate EAGLE3 and draft/target speculation to Drafter (#6007 ) Signed-off-by: ziyixiong-nv <fxiong@nvidia.com>	2025-07-17 21:15:01 +08:00
Enwei Zhu	21efb50068	[TRTLLM-6406] feat: Enable guided decoding with overlap scheduler (#6000 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-17 17:46:10 +08:00
Chuang Zhu	44c70c88f9	chore:[BREAKING CHANGE] use cacheTransceiverConfig as knobs for disagg service (#5234 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-07-17 17:42:07 +08:00
Iman Tabrizian	d4d21a106e	[fix] Release slots with spec decode + disagg (#5975 ) (#6032 ) Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com> Signed-off-by: Iman Tabrizian <10105175+Tabrizian@users.noreply.github.com> Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-17 12:58:18 +08:00
Shiyu Li	6e1aee6fd6	[fix] Performance Optimization for MNNVL TwoShot Kernel (#5934 ) Signed-off-by: Shiyu Li <shili@nvidia.com> Co-authored-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-07-17 10:49:51 +08:00
chenfeiz0326	fe070a0168	test: Update Llama4 Scout FP4 & FP8 accuracy tests (#5901 ) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>	2025-07-17 09:41:18 +08:00
Wanli Jiang	2d2b8bae32	feat: TRTLLM-5574 Add phi-4-multimodal pytorch-backend support (#5644 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-17 06:30:58 +08:00
qixiang-99	e09e409dfb	Fix: Enhance ModelConfig for kv cache size calculations (#5868 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-07-16 14:41:31 -07:00
Mike Iovine	fa34cb7234	[refactor] Clean up drafter/resource manager creation logic (#5805 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-16 12:45:46 -07:00
shaharmor98	e0836f9ca9	[TRTLLM-5493] Add core infrastructure to enable loading of custom checkpoint formats (#5372 ) Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-07-17 00:50:30 +08:00
Wanli Jiang	9354114f68	fix: Update trtllm args issues with extra nested config (#5996 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-16 12:41:45 -04:00
Bo Li	fc2347eaf5	chore: Cleanup disable_fp4_allgather. (#6006 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-07-16 17:54:36 +08:00
Yan Chunwei	a02606a9e2	[TRTLLM-5530][BREAKING CHANGE] refactor: unify KvCacheConfig in LLM class for pytorch backend (#5752 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-16 16:42:59 +08:00
Yan Chunwei	7568deb2f1	[nvbug/5387226] chore: add propogation for trust_remote_code to AutoConfig (#6001 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-16 16:05:38 +08:00
Yiqing Yan	e51c541617	chore: Bump version to 1.0.0rc4 (#6086 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-16 13:02:23 +08:00
Wanli Jiang	8679a058a3	fix: Unable to load phi4-model with tp_size>1 (#5962 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-16 11:39:41 +08:00
danielafrimi	edab7532dd	feat/add latency support for trtllm bench (#3730 ) Signed-off-by: Ubuntu <dafrimi@nvidia.com> Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com> Signed-off-by: Frank <3429989+FrankD412@users.noreply.github.com> Co-authored-by: Daniel Afrimi <dafrimi@nvidia.com> Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com>	2025-07-15 13:13:49 -07:00
Fanrong Li	7a1af1c738	Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/5947 (#5989 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-07-16 01:33:12 +09:00
Xiaodong (Vincent) Huang	0523f77b36	support TRTLLM_DEEP_EP_TOKEN_LIMIT to allow run deep-ep on memory-con… (#5684 ) Signed-off-by: Vincent Huang <vincenth@nvidia.com>	2025-07-15 18:34:21 +03:00
Tailing Yuan	4a26bd6500	Fix: pad DeepEP fp4 recv tensors if empty (#6048 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-07-15 23:14:01 +09:00
MinaHuai	9ebc3ab9c4	[nvbugs/5385972][nvbugs/5387423][Fix] Minor fix for llava_next/llava_onevision (#5998 ) Signed-off-by: Mina Huai <121143971+MinaHuai@users.noreply.github.com>	2025-07-15 10:01:35 -04:00
Jaedeok Kim	ab1c54709d	fix: adjust window sizes of VSWA at torch backend (#5880 ) Signed-off-by: Jaedeok Kim <jaedeokk@nvidia.com>	2025-07-15 17:41:54 +08:00
nv-guomingz	4e4d18826f	chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… (#6003 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-15 15:50:03 +09:00
Lucas Liebenwein	e499f6c44a	[Fix] check for ImportError or ModuleNotFoundError for deep_ep_utils (#6026 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-07-15 14:31:35 +09:00
Rashid Kaleem	2ea4077993	[Model load] Fix llama min-latency model load (#5883 ) Signed-off-by: Rashid Kaleem <4079439+arekay@users.noreply.github.com>	2025-07-15 09:29:19 +08:00
ixlmar	f225f5cd2e	[nvbugs-5318143] fix: restrict PyTorch memory usage to avoid OOMs (#5964 ) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>	2025-07-15 06:49:42 +08:00
brb-nv	f5f5be9e94	enh: Bidirectional mask with multiple images for Gemma3 (#5976 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-14 22:39:18 +08:00
brb-nv	1a2d96919c	feat: Update Gemma3 Vision Encoder (#5973 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-14 22:38:10 +08:00
Yechan Kim	63139fdcff	feat: EXAONE4.0 support (#5696 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-14 22:28:10 +09:00
Zhenhuan Chen	30608a5e6d	[https://nvbugs/5355316 ] fix: update torch.compile option to fix triton store_cubin error (#5865 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-14 17:17:30 +08:00
Robin Kobus	5a61d64b5b	[nvbugs/5345391] fix: chunked prefill + overlap scheduling (#5761 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Pengyun Lin	388b4919b8	[nvbug 5304752][fix] enhance _check_arguments to filter illegal requests for pytorch backend (#5541 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Pengyun Lin	6992616c1f	[nvbug 5004744][fix] rewrite completion API to avoid repetitive tokens (#5201 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Iman Tabrizian	c8874a7f94	[nvbug/5337601][fix] Fix disagg + speculative decoding (#5558 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> Co-authored-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
Dom Brown	afaa388bee	[TRTLLM-6100] fix: Nvbug 5356427: autotuned TRTLLM Gen fp8 block scale MoE illegal memory access (#5676 ) Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-07-14 17:17:30 +08:00
WeiHaocheng	4d8920982a	fix: set allreduce strategy to model config (#5955 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-07-14 17:59:11 +09:00
dominicshanshan	c9e7f831dc	Breaking change: perf: [TRTLLM-4662] Enable cuda graph by default (#5480 ) Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-07-14 16:42:23 +08:00
wili	cfcb97af0e	[BUG5388075][fix] Fix error in post-merge-tests (#5949 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-07-14 14:33:39 +09:00
QI JUN	ce39409530	fix cancel request logic (#5800 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-07-14 10:23:20 +08:00
Mike Iovine	8950223f6f	[fix] Remove SpecConfig and fix thread leak issues (#5931 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-07-12 21:03:24 +09:00
Enwei Zhu	bc1d4fb5da	[NvBug 5378370] fix: Fix alltoall for llama4 (apply_router_weight_on_input=True) (#5902 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-12 15:50:31 +09:00
Thor Johnsen	041f1fa513	[TRTLLM-6264] Fix flaky test_e2e.py::test_openai_lora (#5885 ) Signed-off-by: thorjohnsen <41591019+thorjohnsen@users.noreply.github.com>	2025-07-11 16:20:41 -07:00
2ez4bz	6304866ce8	[refactor] Move vision parts from processor to model for Gemma3 (#5888 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-11 15:13:51 -07:00
brb-nv	0385f89abc	test: Fix Gemma3 unit tests due to transformers upgrade (#5921 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-10 17:24:10 -07:00
Void	854655f2f7	deepEP fp4 post quant all2all dispatch (#5881 ) Signed-off-by: Yilin Zhang <18275976+yilin-void@users.noreply.github.com>	2025-07-11 08:18:54 +08:00
Frank	aa4eebe973	[enhance] Add the ability to write a request timeline. (#5258 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> Signed-off-by: Frank <3429989+FrankD412@users.noreply.github.com>	2025-07-10 17:15:30 -07:00
wili	2e3cf42e03	[refactor] Simplification of Speculative decoding configs (#5639 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-07-10 11:37:30 -04:00
Kaiyu Xie	7b09a415c1	fix: Make the bench serving script compatible with different usages (#5905 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-07-10 19:36:26 +08:00
Enwei Zhu	055c4a9fe6	[NvBug 5370718, 5371538] fix: Fix incremental detokenization (#5825 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-10 16:30:00 +08:00
CarstyYou	dc32f9ae73	[fix] fix tileN cannot % 16==0 & support sm89 deepgemm bmm (#5531 ) Signed-off-by: CarstyYou <186021327+CarstyYou@users.noreply.github.com>	2025-07-10 15:16:18 +08:00
Anthony Chang	7d21b55b5a	[feat] Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE (#5723 ) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com>	2025-07-10 14:06:50 +08:00
Yan Chunwei	07f6da763d	[TRTLLM-5530] chore: rename LLM.autotuner_enabled to enable_autotuner (#5876 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-10 11:31:35 +08:00
Hanjun Cho	6490a27ad7	[feat] Add TensorRT-Engine Qwen3 (dense) model support (#5650 ) Signed-off-by: Ubuntu <ubuntu@ip-10-0-20-146.us-west-2.compute.internal> Signed-off-by: Hanjun Cho <46752251+gkswns0531@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-20-146.us-west-2.compute.internal>	2025-07-10 10:26:06 +08:00
brb-nv	3209b31665	feat: Custom masking utils for Gemma3 VLM (#5853 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-10 06:18:04 +09:00
2ez4bz	87fe44fd29	feat(models): Mistral3.1 VLM pytorch backend support (#5529 ) Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>	2025-07-09 13:17:40 -07:00
Chang Liu	b61a717275	[1/N][TRTLLM-5195][feat] Share PyTorch tensor between processes (#5396 )	2025-07-10 05:12:53 +09:00
Wanli Jiang	3f7cedec7c	Update transformers to 4.53.0 (#5747 ) Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com> Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-09 09:32:24 -07:00
DylanChen-NV	74dca0aa7b	[NVBUG-5304516/5319741]Qwen2.5VL FP8 support (#5029 ) Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>	2025-07-09 23:16:42 +08:00
tomeras91	5aa958a11a	[TRTLLM-5838][fix] fix max batch size and max tokens in kv cache estimations for Nemotron-H (#5371 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-07-09 11:30:15 +03:00
Dom Brown	3e3b1769ad	[TRTLLM-5881] feat: Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner (#5764 ) Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-07-09 08:21:58 +01:00
dongxuy04	dd3c736c7e	chore: some refactor on WideEP (#5727 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-07-09 14:26:57 +08:00
chenfeiz0326	64fd64fcf2	[TRTLLM-6262] Fix Llama4 Scout FP4 crash issue (#5834 ) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>	2025-07-09 14:23:21 +08:00
Chang Liu	4df5f96c8d	[Bugfix] LLama4: fix for llama4 multimodal support (#5809 )	2025-07-09 13:03:40 +09:00
Xianjie Qiao	5ab1cf5ae6	Remove unnecessary benchmarking results (#5852 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-07-09 11:19:06 +08:00
brb-nv	2bd09ed2d4	fix: Skip rope scaling for local layers in Gemma3 VLM (#5857 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-09 10:10:33 +08:00
Omer Ullman Argov	d6d2ab2c99	[fix] Catch inference failures in `trtllm-bench` (#5841 ) Signed-off-by: Omer Ullman Argov <118735753+omera-nv@users.noreply.github.com>	2025-07-09 03:53:03 +03:00
Iman Tabrizian	c508b994b6	Fix lost requests for disaggregated serving (#5815 ) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>	2025-07-09 08:42:45 +09:00
Kaiyu Xie	bb5b16fcb9	feat: Return context response immediately when stream_interval > 1 (#5836 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-07-09 00:19:57 +09:00
Raayan Dhar	e3268a4221	[TRTLLM-5847][feat] Support n-gram speculative decoding with disagg (#5732 ) Signed-off-by: raayandhar <rdhar@nvidia.com>	2025-07-08 09:39:58 -04:00
Yukun He	e104f8bbb5	[5305318] fix: Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. (#5801 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-07-08 19:51:05 +08:00
Yegor	b01d1c28f7	[feat] Detokenize option in /v1/completions request (#5382 ) Signed-off-by: Yegor <75512761+Wokzy@users.noreply.github.com> Signed-off-by: Yegor Yershov <yegor6741@gmail.com>	2025-07-08 19:36:04 +08:00
xiweny	eaf8bec88b	fix: Disaggregate serving with attention DP (#4993 ) Signed-off-by: Xiwen Yu <13230610+VALLIS-NERIA@users.noreply.github.com>	2025-07-08 16:15:03 +08:00
Yiqing Yan	5203a0f6df	chore: bump version to 1.0.0rc3 (#5819 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-08 16:04:40 +09:00
Zhenhuan Chen	dee6644ed9	feat(scaffolding): add streaming scaffolding_llm.generate_async support (#5345 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-08 15:08:40 +09:00
nv-guomingz	0be41b6524	Revert "chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie…" (#5818 )	2025-07-08 13:15:30 +09:00
Yechan Kim	5bc3a15f10	feat: add MultimodalParams & putting all multimodal params into it and refactor HyperCLOVAX & Qwen2/2.5-VL (#5522 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-07-07 18:03:12 -07:00
nv-guomingz	5a8173c121	chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… (#5795 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-08 08:52:36 +08:00
Robin Kobus	30a19fcf7c	[TRTLLM-6291] feat: Add user-provided speculative decoding support (#5204 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-07 16:30:43 +02:00
Tailing Yuan	85b4a6808d	Refactor: move DeepEP from Docker images to wheel building (#5534 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-07-07 22:57:03 +09:00
Daniel Cámpora	1260e2f33f	feat: Optimize TRTLLM Sampler perf single beam single step (#5550 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-07-07 15:44:47 +02:00
DylanChen-NV	5ca2b9bb15	[TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow (#5615 ) Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>	2025-07-07 18:04:57 +08:00
Yan Chunwei	dfce61f4b9	[TRTLLM-5530][BREAKING CHANGE] refactor: LLM arglist rename mixed_sampler to enable_mixed_sampler (#5751 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-07 17:05:14 +08:00
Zheng Duan	de10774c2e	chore: log stack trace on error in openai server (#5749 ) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com>	2025-07-07 14:54:36 +08:00
Daniel Stokes	ec6c7dff1a	feat: Add support for MXFP8xMXFP4 in pytorch (#5535 ) Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>	2025-07-06 15:32:06 -07:00
Robin Kobus	ae27261094	refactor: decoding inputs (#5679 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-07-06 08:21:02 +02:00
Xianjie Qiao	b1976c2add	Add wide-ep benchmarking scripts (#5760 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> Signed-off-by: Xianjie Qiao <5410381+qiaoxj07@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-05 19:29:39 +08:00
Xianjie Qiao	089fd55eda	Add dummy all_reduce for kernel breakdown (#5745 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-07-05 13:08:58 +09:00
Frank	d61893dc77	[fix] Update to properly set cuda graphs in trtllm-bench overrides. (#5634 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-05 05:19:16 +09:00
Stefan Niebler	d1112aac37	[TRTLLM-3442] feat: added beam search support to the PyTorch Workflow (#5333 ) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com>	2025-07-05 01:35:13 +09:00
HuiGao-NV	3ed3bbcb5d	Fix: pass allreduce strategy to pytorchConfig (#5746 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-07-04 21:32:13 +09:00
Shunkangz	32339d1b20	Raise shut down error for each request (#4936 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-07-04 18:58:24 +09:00
Tailing Yuan	e134a52e07	Perf: reduce DeepEPLowLatency memory and time (#5712 ) Signed-off-by: Tailing Yuan <yuantailing@gmail.com>	2025-07-04 14:46:28 +08:00
Shunkangz	a79d8c9f5e	Fix none response in PD (#5422 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-07-04 14:25:10 +08:00
brb-nv	cdaa6abce7	fix: Investigate Gemma3 1B decoder output discrepancy (#5564 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-04 13:14:13 +08:00
Frank	819ae903de	[https://nvbugspro.nvidia.com/bug/5351333 ][fix] Update to chunking calculation. (#5625 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-07-04 13:14:13 +08:00
Clay	7a319524da	feat: support more parameters in openai worker of scaffolding (#5115 ) Signed-off-by: Clay <ccs96307@gmail.com>	2025-07-04 09:35:34 +08:00
Lucas Liebenwein	24ac9b5f69	[AutoDeploy] merge feat/ad-2025-06-29 (#5737 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Co-authored-by: Neta Zmora <nzmora@nvidia.com> Co-authored-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>	2025-07-04 10:21:18 +09:00
Netanel Haber	aa72d39b72	MTP and derivatives: Align sample state with trtllm sampler sample state (#5675 ) This PR moves MTPSampler and derivatives to use the universal seq_slot indexing for sampling. This is the last piece of the puzzle: After this, all of the samplers will use this format. See: `6ee94c7` Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-07-03 19:55:48 +02:00
Zhenhuan Chen	528ff52ef4	[https://nvbugs/5365714 ] fix(scaffolding): use default LLM rather than trt backend LLM (#5705 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-07-03 23:54:20 +09:00
Rashid Kaleem	2b0c87e613	[ModelLoad] Concurrent load model (#5291 ) Signed-off-by: Rashid K <rkaleem@nvidia.com> Co-authored-by: Zhihan Jiang <68881590+nvzhihanj@users.noreply.github.com>	2025-07-03 22:18:04 +08:00
nv-guomingz	8dad22cbe7	chore: refine the default value by using pydantic default instead of … (#5695 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-07-03 22:41:29 +09:00
tomeras91	7dbecf7272	[TRTLLM-4923][feat] Enable CUDA graphs for Nemotron-H (#5646 ) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>	2025-07-03 11:07:51 +03:00
Yiqing Yan	3c9dd5cd66	chore: bump version to 1.0.0rc2 (#5645 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-07-03 12:35:28 +08:00
Enwei Zhu	3a46cf275b	fix: Fix missing arg to alltoall_prepare_maybe_dispatch (#5669 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-07-02 21:41:55 -04:00
Jhao-Ting Chen	77082cde38	[https://nvbugspro.nvidia.com/bug/5329655 ] [feat] Pytorch path add spec dec param to attention op (#5146 ) Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>	2025-07-02 04:54:43 -04:00
qixiang-99	ca7b6ec8d8	Feat/pytorch vswa kvcachemanager (#5151 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-07-02 15:58:00 +08:00
Yan Chunwei	2d69b55fe8	chore: enhance yaml loading arbitrary options in LlmArgs (#5610 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-07-02 14:21:37 +08:00
HuiGao-NV	10c50515c2	fix: Add back allreduce_strategy parameter into TorchLlmArgs (#5637 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-07-02 09:49:20 +08:00
Perkz Zheng	ba2ab5098b	[Bug] attention DP doesn't work with embedding TP (#5642 ) Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com>	2025-07-02 08:57:46 +08:00
Aurelien Chartier	efef911f5e	fix: add missing self. from PR #5346 (#5653 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-07-01 20:38:55 -04:00
Aurelien Chartier	fa95e402a5	feat: add LLmArgs option to force using dynamic quantization (#5346 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-07-01 12:16:09 -07:00
liji-nv	c345f5876c	[feat] Support torch compile for attention dp (#5086 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-07-01 13:48:52 -04:00
Kaiyu Xie	f9a455651b	perf: Use tokenizers API to optimize incremental detokenization perf (#5574 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-07-01 09:35:25 -04:00
Anurag Mukkara	93edfea2b8	[nvbug/5354825] Fix nougat test image url (#5496 ) Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com>	2025-07-01 20:12:55 +08:00
Wanli Jiang	3789ba1d37	feat: TRTLLM-5941 Upgrade xgrammar to 0.1.18 (#5364 ) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>	2025-07-01 20:12:55 +08:00
brb-nv	4ef60d5fbb	nvbugs-5331031; nvbugs-5344203 - address intermittent issues with Mistral Small multimodal for BS=8 (#5453 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-07-01 20:12:55 +08:00
danielafrimi	7a617ad1fe	feat: W4A16 GEMM (#4232 ) Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>	2025-07-01 10:36:05 +03:00
Netanel Haber	6ee94c7ac8	Reintroduce with perf fixes: feature: unify new_tokens format sample state to trtllm samper tokens format (#5513 ) `58a8a8f` - these changes were previously merged to main here. `6aef149` - the changes were temporarily reverted in main, due to a significant perf regression in models using the TorchSampler (observed by @byshiue). This PR is meant to re-merge these changes along with a fix to prevent the regression. The first commit of this PR is actually just the reverted revert - filter it out of the changes to see previously unmerged changes. Signed-off-by: Netanel Haber <nhaber@nvidia.com>	2025-06-30 11:58:59 -07:00
Wei-Ming Chen	f28cd3056e	feat: AutoDeploy fp8 quantization support for bmm (#3849 ) Signed-off-by: Wei-Ming Chen <17592131+meenchen@users.noreply.github.com>	2025-06-30 12:36:34 -04:00
nv-guomingz	6e48ac25a6	chore: remove cuda_graph_ prefix from cuda_graph_config filed members. (#5585 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-06-30 12:23:14 -04:00
Li Min	16fc99391f	refactor: [TRTLLM-6150] Refactor moe permute and finalize op by removing duplicated code (#5557 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-06-30 08:48:04 -07:00
Yan Chunwei	98a7c24062	chore [TRTLLM-6009]: remove ptuning knobs from TorchLlmArgs (#5595 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-30 20:40:23 +08:00
Robin Kobus	9bdc5951f8	refactor: decoder state setup (#5093 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-30 11:09:43 +02:00
Fanrong Li	6cbc9a5297	[nvbug/5354946][fix] Fix mtp vanilla draft inputs (#5568 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-30 15:59:12 +08:00
WeiHaocheng	42a9385d02	[TRTLLM-5331] perf: Replace allgaher with AllToAllPrepare (#5570 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-06-30 13:06:09 +08:00
dongjiyingdjy	852b79053d	feat : support duplicate_kv_weight for qwen3 blockwise scale (#5459 ) Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>	2025-06-30 11:49:22 +08:00
nv-guomingz	578430e64c	[TRTLLM-5530][BREAKING CHANGE]: enhance the llm args pytorch config part 1(cuda_graph_config) (#5014 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-06-30 11:05:40 +08:00
Bo Li	6000380a0c	perf: Avoid reswizzle_sf after allgather. (#5504 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-06-29 21:25:50 +08:00
Talor Abramovich	70e34a3291	[TRTLLM-5831][feat] Add LoRA support for pytorch backend in trtllm-serve (#5376 ) Signed-off-by: Talor Abramovich <talora@nvidia.com>	2025-06-29 12:46:30 +00:00
amirkl94	de9779900c	feat: Add support for YARN in NemotronNAS models (#4906 ) Signed-off-by: Amir Klein <203507526+amirkl94@users.noreply.github.com>	2025-06-29 09:45:49 +03:00
Lucas Liebenwein	619709fc33	[AutoDeploy] merge feat/ad-2025-06-13 (#5556 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-06-29 03:52:14 +08:00
Li Min	6021a439ab	Make moe permute and final as custom op (#5412 ) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com>	2025-06-27 15:48:33 -07:00
Darragh Hanley	5437075def	ReDrafter support for Qwen (#4875 ) Signed-off-by: darraghdog <darragh.hanley@gmail.com> Signed-off-by: Darragh Hanley <darragh.hanley@gmail.com> Co-authored-by: rakib-hasan <rhasan@nvidia.com>	2025-06-28 02:33:10 +08:00
Robin Kobus	a8141a4513	refactor: Speculative decoding buffers part 2 (#5316 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-27 17:41:48 +02:00
Aurelien Chartier	833c0dea4a	[TRTLLM-6104] feat: add request_perf_metrics to LLMAPI (#5497 ) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>	2025-06-27 17:03:05 +02:00
wili	56cdfe5c6c	[TRTLLM-5000][feat] NGrams V2 (#4569 ) Signed-off-by: wili-65535 <wili-65535@users.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@users.noreply.github.com>	2025-06-27 23:00:17 +08:00
peaceh-nv	cb58073ab7	Fix : fix build for sm120 (#5265 ) Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>	2025-06-27 20:42:47 +08:00
Daniel Cámpora	73b8a95049	feat: Use inference mode in update_requests to improve perf of TRTLLM Sampler (#5538 )	2025-06-27 18:40:53 +08:00
Daniel Stokes	83a1f60556	feat: Expose bias and FP8_MXFP4 MOE CUTLASS backend features to pytorch (#5410 ) Signed-off-by: Daniel Stokes <40156487+djns99@users.noreply.github.com>	2025-06-27 12:29:34 +08:00
Yuxian Qiu	dc36228f52	fix: Fix block scale fp8 support for deepseek v3 on Blackwell. (#5514 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-06-27 11:03:38 +08:00
Yibin Li	0f3bd7800e	[TRTLLM-4971]: Use safe deserialization in ParallelConfig (#4630 ) Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>	2025-06-27 09:58:41 +08:00
Frank	aa6e015ef8	Update trtllm-bench to support new Pytorch default. (#5491 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-06-26 17:05:43 -07:00
Venky	0083228d2a	fix: Mapping rank boundary check bug (#4935 ) Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>	2025-06-27 07:27:59 +08:00
jmydurant	8836990bde	[TRTLLM-3602][feat] support nvfp4 model and fp8 kv cache for MLA chunked prefill (Blackwell) (#5475 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-06-26 22:18:08 +08:00
Robin Kobus	8dfa31c71d	refactor: remove batch_manager::KvCacheConfig and use executor::KvCacheConfig instead (#5384 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-26 19:45:52 +08:00
Rashid Kaleem	3a1f4d4001	[feat] Add progress bar to benchmark (#5173 ) Signed-off-by: Rashid Kaleem <rkaleem@nvidia.com> Co-authored-by: Zhihan Jiang <68881590+nvzhihanj@users.noreply.github.com> Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com>	2025-06-26 18:39:45 +08:00
Kaiyu Xie	2eb6502b1d	feat: Add support for TRTLLM CustomDataset (#5511 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-06-26 18:27:37 +08:00
Bo Li	1bab9000a6	perf: Optimize swizzle_sf, unswizzle_sf, reswizzle_sf (#5318 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>	2025-06-26 14:03:56 +08:00
dongxuy04	490d2e5819	feat: large-scale EP(part 8: Online EP load balancer integration for PCIe fp8) (#5226 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-06-25 22:25:13 -07:00
amitz-nv	e0bb123ae7	[TRTLLM-5921][feat] Prevent serialization of entire LoRA adapters in each request (#5080 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-06-26 08:15:06 +03:00
Yukun He	9ee33605bb	[TRTLLM-6019] feat: Remove cutlass min latency code from AutoTuner. (#5394 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-06-26 13:12:03 +08:00
Netanel Haber	6aef14943c	Revert "feature: unify new_tokens format sample state to trtllm samper new_tokens format (#4401 )" (#5474 ) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-06-25 20:56:04 -07:00
jmydurant	578dbc8d9a	feat: chunked prefill for MLA (Blackwell) (#4651 ) Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-06-26 09:01:00 +08:00
Yukun He	3fc57543e2	[5356427] fix: Remove the seq_len of 4096 from FP8 block scale MoE tuning configs. (#5485 ) The seq_len of 4096 will cause some unknown CUDA illegal memory access issue if run with some other tests consecutively. Put a saturated upper bound for any sequence length larger than it.	2025-06-26 08:38:35 +08:00
Xianjie Qiao	1e4fa13d33	Add sleep function for disagg gen-only benchmarking (#5398 ) Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-06-26 07:32:16 +08:00
QI JUN	3a2c4ca77b	chore: split _build_model method for TorchLlm and TrtLlm (#5418 ) Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-06-26 04:32:46 +08:00
Mike Iovine	5bc8c894f7	[chore] Disable block reuse when draft model speculation is being used (#5448 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-06-26 03:51:20 +08:00
Daniel Cámpora	205c97a4ae	[TRTLLM-5974][feat] Support disaggregated serving in TRTLLM Sampler (#5328 ) Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> Signed-off-by: Daniel Cámpora <961215+dcampora@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-06-25 17:41:36 +02:00
Kaiyu Xie	c5ae3272b9	feat: Make benchmark_serving part of the library (#5428 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-06-25 23:13:56 +08:00
HuiGao-NV	b3a4c1f404	feat: Remove not used padding_idx in models (#5385 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-06-25 17:19:59 +08:00
Yiqing Yan	f3cfe86dd1	chore: bump version to 1.0.0rc1 (#5460 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-06-25 16:21:34 +08:00
Enwei Zhu	fc7a81ceb0	test: Add LLGuidance test and refine guided decoding (#5348 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-25 14:12:56 +08:00
Enwei Zhu	76da7fed86	fix (NvBug 5354925): Fix static EPLB (#5411 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-25 13:14:40 +08:00
Shunkangz	d5354897c0	feat: Dynamically remove servers in PD (#5270 ) Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-06-25 09:50:04 +08:00
Lucas Liebenwein	5cffb7e0ec	[AutoDeploy] Merge feat/ad_2025_06_13 feature branch (#5454 ) Signed-off-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com> Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Co-authored-by: Grzegorz Kwasniewski <213329731+greg-kwasniewski1@users.noreply.github.com> Co-authored-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Co-authored-by: Frida Hou <201670829+Fridah-nv@users.noreply.github.com>	2025-06-25 09:30:13 +08:00
bhsueh_NV	73ba4fc320	fix: fix bug of qwen3 + eagle3 + finalize_moe_fusion (#5369 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-06-25 09:20:23 +08:00
dongxuy04	699520082b	Add MTP support for Online EPLB (#5213 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-06-25 07:58:13 +08:00
QI JUN	d93a5e04b5	Chore: remove unused variables (#5314 ) Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-06-24 22:27:32 +08:00
HuiGao-NV	35a92f6bab	Add debug hook to support dump tensor data and add new debug functions easily (#5182 ) Signed-off-by: Hui Gao	2025-06-24 17:45:28 +08:00
Luis Vega	d26040e5d9	chore: delete mamba hybrid, since it is now called NemotronH (#5409 ) Signed-off-by: Luis Vega <vegaluisjose@users.noreply.github.com>	2025-06-24 16:27:31 +08:00
Robin Kobus	e2a8cbc80b	refactor: manage cache indirection in decoder state (#5315 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-24 09:15:59 +02:00
HuiGao-NV	e16c1bef6e	[fix] Add 1 and draft_token_num to seq_len when overlap scheduling is enabled during memory estimation (#5343 ) Signed-off-by: Hui Gao <huig@nvidia.com>	2025-06-24 11:43:43 +08:00
Netanel Haber	58a8a8fd37	feature: unify new_tokens format sample state to trtllm sampler new_tokens format (#4401 ) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-06-23 10:38:37 -07:00
dongxuy04	4f0f17ac8a	feat: Misc Opt for large scale EP (#5374 ) Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>	2025-06-20 13:11:31 +08:00
Fanrong Li	5d4ab47d5b	fix: refactor and fix mtp vanilla (#4762 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-20 05:23:39 +08:00
Yan Chunwei	9bd42ecf9b	[TRTLLM-5208][BREAKING CHANGE] chore: make pytorch LLM the default (#5312 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-20 03:01:10 +08:00
Kaiyu Xie	7246fd75d1	feat: Support stream_interval (#5284 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-06-19 21:57:10 +08:00
Fanrong Li	c7af650d5a	Fix: fix the deterministic issue in the MTP Eagle path (#5285 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-19 18:08:40 +08:00
Frank	68687a9f56	[WAR][nvbug/5321947] Add an async sleep to unblock event loop. (#5342 ) Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>	2025-06-19 17:25:18 +08:00
hlu1	b558232ce1	Refactor CutlassFusedMoE (#5344 ) Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>	2025-06-19 00:04:07 -07:00
amitz-nv	1753202b61	[TRTLLM-5825][fix] Fix torch LoRA TP (#5338 ) Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-06-19 09:12:00 +03:00
Yiqing Yan	dedce8ab0e	chore: bump version to 1.0.0rc0 (#5326 ) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>	2025-06-19 12:02:28 +08:00
nv-guomingz	6a388b105a	chore: remove torch_compile prefix for TorchCompileConfig field members (#5261 ) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>	2025-06-19 09:21:51 +08:00
Zongfei Jing	2b23cd56ce	[feat] Fusion finalize and allreduce for qwenmoe model (#5223 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> Co-authored-by: Kefeng-Duan <176893526+Kefeng-Duan@users.noreply.github.com>	2025-06-19 08:03:58 +08:00
Yan Chunwei	3946e798db	fix[nvbug5298640]: trtllm-llmapi-launch multiple LLM instances (#4727 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-19 06:13:53 +08:00
jellysnack	0623ffe3bc	feat: Add LLGuidance Support for PyTorch Backend (#5214 ) Signed-off-by: jellysnack <oleg.jellysnack@gmail.com> Signed-off-by: jellysnack <158609015+jellysnack@users.noreply.github.com> Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-18 19:33:34 +08:00
Zhanrui Sun	516bd4dc05	chore: bump version to 0.21.0rc3 (#5309 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-06-18 15:59:53 +08:00
Robin Kobus	38547b92f3	refactor: Introduce ResourceManagerType enum for resource management (#5246 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-18 09:55:59 +02:00
Yukun He	6711ad9cf3	[TRTLLM-5589] feat: Minor optimizations for tunable FP8 batched GEMM op. (#5139 ) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>	2025-06-18 14:33:46 +08:00
Yan Chunwei	724e495254	chore: partition LLM class into TorchLLM and TrtLLM (#4900 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-06-18 14:01:25 +08:00
Yi Zhang	e44f7687af	feat: Add no_kv_cache_reuse option and streaming support for trtllm serve bench (#4971 ) Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>	2025-06-18 13:37:31 +08:00
QI JUN	855036d8ee	update LlmRequest.is_dummy property (#5283 ) Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-06-18 10:52:13 +08:00
Robin Kobus	627062c265	refactor: Update decoder buffer and logits management (#4450 ) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>	2025-06-18 08:10:32 +08:00
Mike Iovine	9bf69c9fdb	[chore] Remove BaseDraftTokenManager (#5251 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-06-17 11:57:52 -04:00
QI JUN	f899c4d294	Re-implement LlmResponse in Python to reduce host overhead of pybind (#5224 ) Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-06-17 21:28:09 +08:00
Dom Brown	44fb3c1673	[TRTLLM-5770] feat: Integrate TRT-LLM Gen FP8 block scale MoE with Pytorch workflow kernel autotuner (#5207 ) - Adds a new Python custom op (fp8_block_scale_moe_runner) and a FP8BlockScaleMoERunner class for autotuning. - Updates C++ MoE and batched GEMM kernels to accept a configIndex for workspace sizing and execution. - Extends the unit test to run both autotuned and non-autotuned code paths. Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-06-17 21:01:56 +08:00
amirkl94	8451a87742	chore: Mass integration of release/0.20 (#5082 ) Signed-off-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com> Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Co-authored-by: Stanley Sun <190317771+StanleySun639@users.noreply.github.com> Co-authored-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com> Co-authored-by: Erin <14718778+hchings@users.noreply.github.com> Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com> Co-authored-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Co-authored-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>	2025-06-17 14:32:02 +03:00
liji-nv	13eef642e6	[feat] Piecewise cuda graph support for MLA (#4467 ) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>	2025-06-17 18:58:38 +08:00
Yilin Fan	498fadceb4	[feat] Add EAGLE3 support for Qwen3 (#5206 ) Signed-off-by: Yilin Fan <206948969+nv-yilinf@users.noreply.github.com>	2025-06-17 17:07:06 +08:00
Enwei Zhu	4b82b8b4c7	[TRTLLM-5330] perf: Optimize MoE supplementary kernels for large-scale EP (#5215 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-06-17 15:23:24 +08:00

... 6 7 8 9 10 ...

1375 Commits