TensorRT-LLMs

mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00

Author	SHA1	Message	Date
Zhanrui Sun	f2c0565577	chore: bump version to 0.21.0rc0 (#4465 ) * chore: bump version to 0.21.0rc0 Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> * Update CODEOWNERS Signed-off-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com> --------- Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Signed-off-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-05-20 12:19:50 +08:00
Lucas Liebenwein	de409e8468	[AutoDeploy] HF factory improvements (#4371 ) * [AutoDeploy] HF factory improvements Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * improve monkey-patches and add unit tests Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-05-19 20:13:43 -07:00
Zhenhuan Chen	e70a205dab	[TRTLLM-4638] feat(scaffolding): update Reward Controller to PRM specific controller with step split (#4337 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-05-19 17:53:41 +08:00
Yuxian Qiu	cf6cd940e5	feat: Add pp support for hybrid attn/mamba model (#4358 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-05-19 14:47:45 +08:00
Netanel Haber	9cd8148f28	API Breaking Change + Readability: "decoder"->"sampler" (#4121 ) * decoder->sampler; new_tensors_device: dict[str, torch.Tensor] -> device: SampleStateTensors * Breaking Change, as it changes public interfaces, main changes: * PyTorchConfig [consumed via LLM(pytorch_backend_config)]: Configuration parameters mixed_decoder and enable_trtllm_decoder -> sampler. * Command-line argument --enable_trtllm_decoder becomes --enable_trtllm_sampler in examples/pytorch/quickstart_advanced.py. --------- Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>	2025-05-16 23:52:25 +08:00
Lucas Liebenwein	8e4320ede5	[AutoDeploy] configurable cache resize (#4372 ) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-05-16 10:07:09 -04:00
Daniel Cámpora	df19430629	chore: Mass Integration 0.19 (#4255 ) * fix: Fix/fused moe 0.19 (#3799) * fix bug of stream init Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * fix bug Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> --------- Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * fix: Add pre-download of checkpoint before benchmark. (#3772) * Add pre-download of checkpoint before benchmark. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * Add missing remote code flag. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * Move from_pretrained to throughput benchmark. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * Move download and use snapshot_download. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * Removed trusted flag. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * Fix benchmark command in iteration log test. Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> --------- Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> * [https://nvbugspro.nvidia.com/bug/5241495][fix] CUDA Graph padding with overlap scheduler (#3839) * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fuse Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * TRTLLM-4875 feat: Add version switcher to doc (#3871) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> * waive a test (#3897) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * docs:fix https://nvbugs/5244616 by removing new invalid links. (#3939) Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com> Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com> * fix: remote mpi session abort (#3884) * fix remote mpi session Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * fix Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> --------- Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * skip fp8 gemm for pre-hopper (#3931) Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> * [https://nvbugspro.nvidia.com/bug/5247148][fix] Attention DP with overlap scheduler (#3975) * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * update multigpu list Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix namings Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * Doc: Fix H200 DeepSeek R1 perf doc (#4006) * fix doc Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com> * update perf number Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com> --------- Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com> * Fix the perf regression caused by insufficient cache warmup. (#4042) Force tuning up to 8192 sequence length for NVFP4 linear op. Also, make this runtime-selectable with UB enabled. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> * doc: Update 0.19.0 release notes (#3976) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> * Optimize the AutoTuner cache access code to reduce host code overhead. (#4060) The NVFP4 Linear op is very sensitive to the host overhead. This PR introduces customizable `find_nearest_profile` and `get_cache_key_specifc`, which allow users to override the default method for generating the cache key. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> * Update switcher (#4098) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> * doc: update release notes (#4108) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> * docs:update 0.19 doc. (#4120) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> * docs:add torch flow supported model list. (#4129) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> * doc: Release V0.19 Perf Overview Update (#4166) Signed-off-by: zpatel <22306219+zbpatel@users.noreply.github.com> * Fix readme of autodeploy. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update tensorrt_llm/_torch/pyexecutor/llm_request.py Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Signed-off-by: Daniel Cámpora <961215+dcampora@users.noreply.github.com> * Revert mgmn worker node. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Change to disable_overlap_scheduler. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> --------- Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com> Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com> Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Signed-off-by: zpatel <22306219+zbpatel@users.noreply.github.com> Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> Signed-off-by: Daniel Cámpora <961215+dcampora@users.noreply.github.com> Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com> Co-authored-by: Frank <3429989+FrankD412@users.noreply.github.com> Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Co-authored-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Co-authored-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com> Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com> Co-authored-by: Yukun He <23156053+hyukn@users.noreply.github.com> Co-authored-by: Zac Patel <22306219+zbpatel@users.noreply.github.com>	2025-05-16 10:53:25 +02:00
Barry Kang	0e14941b7f	[fix] Fixed incorrect mixed precision MoE conversion (#4351 ) Fix for mixed precision MoE conversion Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>	2025-05-16 13:43:41 +08:00
WeiHaocheng	54d28718c7	feat: support benchmark on scaffolding (#3328 ) (#4286 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>	2025-05-16 12:28:49 +08:00
Suyog Gupta	b0f7522c82	[AutoDeploy]feat: Add an AutoDeploy compile backend that only calls torch.compile (#4240 ) * add a torch-compile backend Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * readme changes Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * plumb torch-compile through build_and_run_ad.py Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * plumb torch-compile through build_and_run_ad.py Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * plumb torch-compile through build_and_run_ad.py Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * add torch-cudagraph backend Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * update readme Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * update readme Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> * further enhanced compiler backends Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * further enhance readme Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * better specified defaults in simple_config.py Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * fix typo in simple_config.py Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * updated deepseek-v3 support Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * revert accidental deletion in AD Readme Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Co-authored-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-05-16 08:38:15 +08:00
Yechan Kim	c6e2111f4e	feat: enhance trtllm serve multimodal (#3757 ) * feat: enhance trtllm serve multimodal 1. made the load_image and load_video asynchronous 2. add image_encoded input support to be compatible with genai-perf 3. support text-only on multimodal mdoels(currently, Qwen2-VL & Qwen2.5-VL) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * add test Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix bandit Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * trimming uils Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * trimming for test Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * genai perf command fix Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * command fix Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * refactor chat_utils Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * stress test genai-perf command Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> --------- Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-05-15 16:16:31 -07:00
yuxianq	4f8afe4cc6	feat: [nvbugs/5261055][nvbugs/5170160] non-invasive pipeline parallelism (#4034 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-05-16 04:16:53 +08:00
yuxianq	0e87fcc228	refactor: use x is None instead of x == None. (#4244 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-05-15 20:00:04 +08:00
zhhuang-nv	97bc680cd8	feat: support kv cache reuse for MLA (#3571 ) * support kv cache reuse for MLA load compressed_kv and k_pe and do up-projection use 192/128 head size MLA context kernel support Blackwell and Hopper now Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * add CI test Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * fix: set k_pe head_num to 1 for kernel 2 and kernel 2V2 Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com> * resolve comments Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * use GPTJ style RoPE for MLA Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * fix rebase error and some docs Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * fix kv_lens Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * tiny fix Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * fix torch compile Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * fix: use normal device memory instead of pinned memory for unit test Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com> * fix L0 tests Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * fix torch compile after rebase Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * resolve comments Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> * resolve comments again Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> --------- Signed-off-by: Zhen Huang <145532724+zhhuang-nv@users.noreply.github.com> Signed-off-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com> Signed-off-by: zhhuang-nv <145532724+zhhuang-nv@users.noreply.github.com> Co-authored-by: Mingyang Jiang <13463932+jmydurant@users.noreply.github.com>	2025-05-15 15:22:21 +08:00
Kaiyu Xie	b4e5df0ee0	Breaking change: perf: Enable scheduling overlap by default (#4174 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-05-15 14:27:36 +08:00
dominicshanshan	404fbe9b32	[https://nvbugs/5277113 ][fix]genai-perf API change stress test (#4300 ) * fix bug 5277113. Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com> * fix bug 5277113 and 5278517. Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com> --------- Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>	2025-05-15 14:12:34 +08:00
Simeng Liu	efe0972efb	doc: Add tensorrtllm_backend serving documentation in the Deepseek-V3 README (#4338 ) Add tensorrtllm_backend serving option in the Deepseek-V3 README Signed-off-by: Simeng Liu <simengl@nvidia.com>	2025-05-15 09:31:28 +08:00
Kaiyu Xie	6c45586c51	chore: Remove deprecated Python runtime benchmark (#4171 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-05-14 18:41:05 +08:00
Barry Kang	20b42912ce	[TRTLLM-3330][feat] Support DeepSeek-R1 W4A8 on Hopper (#4123 ) Support DeepSeek-R1 W4A8 on Hopper Co-authored-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> Co-authored-by: Jiang Shao <91270701+StudyingShao@users.noreply.github.com> Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>	2025-05-14 15:48:07 +08:00
brb-nv	8280c3d4f2	feat: Support Gemma3-1b-it in Pytorch workflow (#3999 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-05-14 14:02:44 +08:00
Zhanrui Sun	23b9705bf4	chore: bump version to 0.20.0rc3 (#4261 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-05-14 10:15:25 +08:00
brb-nv	cd5b3d21a0	feat: Support Mistral Small 3.1 24B VLM in TRT workflow (#4183 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-05-14 03:47:22 +08:00
bhsueh_NV	24be357964	doc: update qwen3 document (#4246 ) * update qwen3 document Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * remove wrong note Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> --------- Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-05-13 11:05:47 +08:00
Yechan Kim	3e9bda3a09	[feat] Support HyperCLOVAX-SEED-Text language part (#3902 ) * feat: support HyperCLOVAX-SEED-Text language part Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * add Pytorch flow and remove test file Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * revert summarize Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix summarize Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * remove from pytorch example Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> --------- Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-05-12 16:05:14 +08:00
Zhenhuan Chen	9212e9a740	[TRTLLM-4911] feat(scaffolding): make sampling_params only setable by controller (#4151 ) feat(scaffolding): make sampling_params only setable by controller Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-05-12 15:29:09 +08:00
xinhe-nv	186e2b8c38	[TRTQA-2802][fix]: add --host for mgmn serve examples script (#4175 ) remove prepare data Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>	2025-05-12 13:28:42 +08:00
mayani-nv	fe3a993234	chore: PR to fix the formatting errors (#4200 ) * updating the run_dtm_pld.py to handle logits correctly * following correct code formatting * Update run_dtm_pld.py to account for correct code formatting Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> * correct formatting for the multimodal README PR --------- Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> Co-authored-by: Ubuntu <Azureuser@mayani-nch100-vm1.42fmcfe2wyrepoque0x2bs4uue.jx.internal.cloudapp.net> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-05-09 16:31:36 -07:00
mayani-nv	5c1c69cf9c	fix: draft target README and assertion for logits-based acceptance (#4167 ) * updating the run_dtm_pld.py to handle logits correctly * following correct code formatting * Update run_dtm_pld.py to account for correct code formatting Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> --------- Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> Co-authored-by: Ubuntu <Azureuser@mayani-nch100-vm1.42fmcfe2wyrepoque0x2bs4uue.jx.internal.cloudapp.net> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-05-09 16:08:47 -07:00
mayani-nv	25533a7736	Updating the multimodal models README to add steps for running phi-4-multimodal instruct (#3932 ) * Update run.py for draft_target_model This change makes the draft target model works without mismatch in the vocab size Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> * updating README with phi-4-multimodal-instruct steps * adding ENGINE_DIR, HF_DIR and CKPT_DIR as per review * addressing review comments on PR * updating readme --------- Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> Co-authored-by: rakib-hasan <rhasan@nvidia.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-05-09 15:42:58 -07:00
WeiHaocheng	0f01826dde	feat: support task collection for to collect information (#3328 ) (#3824 ) Signed-off-by: fredw (generated by with_the_same_user script) <20514172+WeiHaocheng@users.noreply.github.com>	2025-05-09 17:09:01 +08:00
Yi Zhang	91bf5e6a8e	[TRTLLM-3105][feat] Add Piecewise CUDA Graph Support (#3804 ) Add Piecewise CUDA Graph Support Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>	2025-05-09 11:04:01 +08:00
shaharmor98	7d94c9561f	feat: support multi lora adapters and TP (#3885 ) * support multi lora, tp Signed-off-by: Shahar Mor <17088876+shaharmor98@users.noreply.github.com>	2025-05-08 23:45:45 +08:00
Enwei Zhu	74df12bbaa	[TRTLLM-4480][doc] Documentation for new accuracy test suite and trtllm-eval (#3946 ) * fix formula Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * update doc Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * 1st version Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * polish Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-05-08 19:35:23 +08:00
Yan Chunwei	389614ca99	chore: remove data stage in serve example on slurm (#4138 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-05-08 11:18:56 +08:00
Ivy Zhang	d7c51c953b	test: add INTEGRATION_TEST env var to speed up integration test (#3618 ) add INTEGRATION_TEST env var Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>	2025-05-08 10:44:50 +08:00
Yan Chunwei	0c26059703	chore: Cleanup deprecated APIs from LLM-API (part 1/2) (#3732 ) * beam_width and max_new_token Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * remove beam_width Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * remove min_length Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * remove return_num_sequences Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> --------- Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-05-07 13:20:25 +08:00
Enwei Zhu	c28b90984f	[TRTLLM-3925, https://nvbugs/5245262 ] [fix] Normalize LLM.generate API (#3985 ) * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-05-07 11:06:23 +08:00
Chuang Zhu	09a28becae	fix cache buffer (#3942 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-05-07 09:49:44 +08:00
bhsueh_NV	5c0f554b9e	doc: update qwen3 document (#4073 ) * update qwen3 document Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * remove useless codes Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> --------- Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-05-06 08:42:51 +08:00
pansicheng	e84dc6b3c7	feat: add deepseek-r1 reasoning parser to trtllm-serve (#3354 ) * add deepseek-r1 reasoning parser Signed-off-by: pansicheng <sicheng.pan.chn@gmail.com> * fix test Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com> --------- Signed-off-by: pansicheng <sicheng.pan.chn@gmail.com> Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com> Co-authored-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-05-06 08:13:04 +08:00
yuxianq	2cfcdbefee	feat: run mmlu and summarize without engine_dir. (#4056 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-05-05 19:35:07 +08:00
Yechan Kim	061a6209b5	fix: [nvbug/5252057] Fix kv cache reuse on PyTorch multimodal (#4025 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>	2025-05-02 10:53:06 -07:00
bhsueh_NV	561ee44737	add ci and doc for qwen3 (#4022 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-05-02 14:13:38 +08:00
Lucas Liebenwein	be916b19e0	feat: [AutoDeploy] unfusing attention for native support (#3668 ) * [AutoDeploy] unfused streamlined attention + caching Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * improved unit testing Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * reviewer feedback Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * some updates to attn_mask handling Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * updated manual benchmarking and cudagraph capture Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-05-02 09:06:49 +08:00
Erin	8fe7bdeacf	feat: LogitsProcessor in PyTorch backend (#3145 ) * support lp in pytorch backend Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> * fix tp Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> --------- Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>	2025-05-01 14:15:30 -07:00
YueWeng	b1621e8d4e	feat: add relaxed acceptance for DS (#3865 ) * add relaxed acceptance for DS R1 Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> * clean and update docs Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> * fix Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> * Modified based on review Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> * fix mtp manager issue Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> --------- Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-05-01 21:50:36 +08:00
Zhanrui Sun	86e7474a9b	chore: bump version to 0.20.0rc2 (#3949 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-30 11:44:43 +08:00
Dom Brown	8709fe8b53	chore: bump version to 0.19.0 (#3598 ) (#3841 ) test: add test cases for 0.19 release (#3608) * fix test name * add quickstart test for nemotron-ultra * add rcca multi-node test case for deepseek-v3 * add rcca info --------- squash (#3642) fix: nvbugs/5187237: fix deterministic mode crash (#3448) * nvbugs/5187237 nvbugs/5112075: fix deterministic mode error * remove waive * Revert "remove waive" This reverts commit 0bf5486d19906d692bfb7a6262333c296b0087ac. * revert ar fusion --------- update fp8 doc (#3647) tests: change qa perf test to trtllm-bench (#3619) fix: FP8 quantized lm_head (NvBug 5214229) (#3567) infra: Add PR approval protection for the release branch (#3634) fix: nvbugs/5231298: pytorch allreduce issue (#3673) Fix: nvbugs/5222698 variable not defined (#3630) * Fix: nvbugs/5222698 variable not defined * Tidy code --------- test:sync waives.txt from main branch by disabling test_perf/gpt_350m-cppmanager case (#3685) test:restore fp8 kv cache testing for L0 (#3671) doc: Update DeepSeek perf docs (#3693) * Update DeepSeek perf docs * update * Apply suggestions from code review --------- tests: waive test_llm_multi_node (#3664) fix: update test_user_buffers_mm_add_prologue atol (#3711) Fix: cherry-pick hmac encryption from main branch (#3635) * security fix cherry-pick changes from main * fix hmac in remote mpi session (#3649) --------- Un-waive DS-V3-Lite tests. (#3621) fix: FP8 kv accuracy (#3675) * fix FP8 kv accuracy * update doc --------- Fix script options for engines. (#3622) unwaive multi-node test (#3721) chore : Split more tests out of gpt tests (#3524) (#3674) doc:add torch examples link into torch backend documentation (#3749) test: Get Eagle tests working (#3593) (#3722) Waive L0 test (#3756) waive failed case in perf test, change default max_batch_size to 512 and write config.json to output log (#3656) Update ds v3 parameters in stress test. (#3676) waive gemma on L20 (#3766) https://nvbugs/5141291: Fix convert.py script for Qwen model. (#3758) Include Qwen2VLDecoderLayer in the smooth_qwen2_model function. fix: PP4 fixes and cleanup (#3688) remove benchmark test list (#3643) skip disagg deepseek test if sm!=90 (#3720) test: skip failed cases on B200 (#3710) * add skip condition to tests * fix error --------- test: [nvbug: 5234494] skip_pre_ada for fp8 cases (#3718) * skip_pre_ada for fp8 cases * update * update after rebase --------- add know issue to deepseek doc. (#3800) Fix ModelOpt Mixtral AWQ OOM (#3714) (#3761) Waive L0 tests (#3826) fix: Reduce memory usage in fused moe op associated with AutoTuning and fix moe fallback issue. (#3793) * Reduce memory usage in fused moe op associated with AutoTuning. * Replace pre-defined bucket size strategy with a generating function based on the tune_max_num_tokens. * Add free_memory logic of workspace in min_latency_mode fused moe path. * Fix fused_moe fallback issue. (#3652) min_latency_mode is only set to False during warmup phase. Thus when it becomes true during inference, all tactics fall back to the default one and thus cause perf regression. --------- [doc] Better document for Draft-Target-Model (DTM) speculative decoding (#3797) Fix pre-commit Fix again Address some review comments for the MI Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com> Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-29 16:57:22 +08:00
Jinyang Yuan	dafc28fb85	fix: Fix FMHA-based MLA in the generation phase and add MLA unit test (#3863 )	2025-04-29 09:09:43 +08:00
Xianjie Qiao	3617e948fd	Add docs about DeepSeek-R1 long context support. (#3910 ) * Add docs about DeepSeek-R1 long context support Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> * update docs Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> * reformat Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com> --------- Signed-off-by: Xianjie <5410381+qiaoxj07@users.noreply.github.com>	2025-04-28 18:33:05 +08:00
Zhenhuan Chen	ad15e45f07	[TRTLLM-4638 ][feat] add best of n support with reward model in scaffolding (#3807 ) Signed-off-by: Zhenhuan Chen <chenzhh3671@gmail.com>	2025-04-28 17:15:33 +08:00
Tao Li @ NVIDIA	2fe35924e3	Fix the link of doc (#3903 ) Signed-off-by: taoli <litaotju@users.noreply.github.com> Co-authored-by: taoli <litaotju@users.noreply.github.com>	2025-04-28 14:41:40 +08:00
milesial	362a8272f8	feat: llama4 input processor (#3383 ) Signed-off-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com> Signed-off-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com> Co-authored-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-04-25 16:47:14 -07:00
Mike Iovine	68e774ff9e	[chore] Add Llama 4 Maverick to quickstart README (#3848 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-04-26 01:04:24 +08:00
rakib-hasan	ccd1eb67ec	Adding local paths to the datasets to make them loadable in offline mode (#3750 ) * adding local paths to the datasets to make them loadable in offline mode Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * bert datasets should work on both offline and online mode Signed-off-by: Rakib Hasan <rhasan@nvidia.com> --------- Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-04-25 03:51:00 +08:00
Luis Vega	f95dbbb6cb	added nemotron-h to supported models (#3663 ) Signed-off-by: Luis Vega <lvega@nvidia.com>	2025-04-24 10:41:32 -07:00
WeiHaocheng	3fc2a16920	feat(part 2): Enhance the integrated robustness of scaffolding with __init__.py #3305 (#3731 ) Signed-off-by: fredw (generated by with_the_same_user script) <20514172+WeiHaocheng@users.noreply.github.com>	2025-04-24 18:47:03 +08:00
Zhanrui Sun	ae34d60108	chore: bump version to 0.20.0rc1 (#3834 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-24 17:43:37 +08:00
qixiang-99	3f67a4c9d8	fix: Set default prompts and media for multimodal quickstart example (#3792 ) Signed-off-by: qixiang-99 <203170375+qixiang-99@users.noreply.github.com>	2025-04-23 22:02:28 -07:00
Mike Iovine	bc5fe7800d	[chore] Fix KV cache block reuse flag name in quickstart_advanced (#3781 ) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>	2025-04-24 06:02:47 +08:00
QI JUN	d0d19e81ca	chore: fix some invalid paths of contrib models (#3818 ) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-04-24 05:36:16 +08:00
Kaiyu Xie	dfbcb543ce	doc: fix path after examples migration (#3814 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-04-24 02:36:45 +08:00
Daniel Cámpora	1299f27c74	fix: Fix C++ decoder synchronization in PyTorch (#3106 ) * Use updateDecoderBuffers in python decoder. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Fix synchronize in trtllm decoder. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Enable by default. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Use guided_decoder to setup seqslots and free them. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Use always decode_async and update_requests. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update decoder buffers. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Fix speculative decoding tests. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Send new_tensors_host instead of assuming dict. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Make default False in enable_trtllm_decoder. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Partially fix mtp, partially fix py_executor. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update request states before sending disagg ctx cache. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Fix disagg test for torch decoder. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Make isend_tensor_list and recv_tensor_list for sending the tensors_host. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Formatting. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Fix rebase. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Add disagg serving case to guided decoder. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Get overlap scheduling to work. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update cutlass to main. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update after rebasing. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Formatting. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update to use decode async and update requests. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Properly pass information to update_requests Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Formatting. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Make disaggregated serving a step closer to working. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Fix rebase. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Fix rebase and format. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Copy new device tokens more pythonic. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Restore MTP add dummy reqs. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Add ordereddict import to py_executor. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Formatting. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Added seq slot manager. Add test. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Use transmission for single tensor except when list of tensors is received. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Add TRTLLMDecoder allocation to estimate max kv cache tokens. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Add stream synchronization Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Formatting. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Make memory calculation of decoder adapt to the chosen decoder. Recognize decoder option passed in executorconfig. Make overlap scheduler test run on TinyLlama. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Format Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Add decoder creation to estimate max kv. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Formatting. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> * Update submodule UCXX inline with main. Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com> --------- Signed-off-by: Daniel Campora <961215+dcampora@users.noreply.github.com>	2025-04-23 23:55:27 +08:00
shaharmor98	49262a62a5	add passing E2E LoRA flow (#3788 ) add passing E2E LoRA flow (#3788) Signed-off-by: Shahar Mor <smor@nvidia.com>	2025-04-23 18:38:06 +03:00
Enwei Zhu	a51b3cf7a6	[TRTLLM-4763][test] Accuracy test improvement (Part 3.6): Deprecate mmlu_llmapi.py (#3802 ) * cleanup mmlu_llmapi.py Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * polish Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-04-23 23:05:13 +08:00
Zongfei Jing	1e5af736ea	Add smart router for moe (#3641 ) Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-04-23 12:21:59 +08:00
rakib-hasan	b16a127026	fixing the metric fmeasure access (#3774 ) Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-04-23 05:10:04 +08:00
Lucas Liebenwein	06b914e0f9	feat: [AutoDeploy] generalizing cudagraph to multiple dynamic inputs (#3589 ) * generalizing cudagraph to multiple dynamic inputs Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> * fix for failing test Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> --------- Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>	2025-04-23 03:38:51 +08:00
rakib-hasan	74c13ea84f	datasets API change : datasets.load_metric => evaluate.load (#3741 ) Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-04-22 08:23:48 +08:00
Enwei Zhu	3fa19ffa4e	test [TRTLLM-4477,TRTLLM-4481]: Accuracy test improvement (Part 3.5): Support GSM8K and GPQA (#3483 ) * add gsm8k Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix gsm8k Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * add gpqa Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * conditional import lm_eval Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * gpqa in lm_eval Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * system prompt Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * shuffle Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * update AA prompt and regex Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * revert AA prompt and regex Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * integration to tests Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * add DS-R1 Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix and clean Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * update tests Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * update Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * clean up Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * free_gpu_memory_fraction=0.8 Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-04-22 07:38:16 +08:00
katec846	eeb605abd6	feat: Offloading Multimodal embedding table to CPU in Chunked Prefill Mode (#3380 ) * Feat: Offload ptable to cpu if enable_chunk_context Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Feat: offload ptable to cpu for chunk context mode Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Fix and add comment Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Update Readme for multimodal and add a new param mm_embedding_offloading Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * fix: Correct prompt table offloading condition in PromptTuningBuffers Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Clean up the code Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Add commits to explain copy from cpu <-> gpu using pinned memory Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Fix namings based on comments Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Fix format based on precommit Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> * Modify --mm_embedding_offloading flag Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> --------- Signed-off-by: Kate Cheng <yunhsuanc@nvidia.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-04-21 14:31:01 +08:00
hlu1	31624b079a	feat: [Deepseek] Add trtllm-gen MOE FP4 MOE backend (#3387 ) * Add TRT-LLM Gen MOE to Deepseek fix fused moe rebase bug. Fix atol in test_fp4_gemm_quantize.py fix fused moe rebase bug. Fix FusedMoe. Disable 2nd routing kernel preexit Bump routing reduction to fp32 Disable PDL for fc1 [DEBUG] Lift token limit to 16k [Bugfix] Token limit to 16k + fp32 routing + tanh Make fp8 tileN 8 Fix FP8 MoE + Remove redundent temp output for FP4 [FP8-only] Avoid wasting CTAs for activation kernel fix: unblock FP8 weightloading with trtllm-gen Remove max_token limit for trtllm-gen path perf: avoid type-conversion and fill_ from aten Minor fix Signed-off-by: Hao Lu <haolu@nvidia.com> * Fix rebase issues Signed-off-by: Hao Lu <haolu@nvidia.com> * Fix compile issue Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> * CI clean Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> --------- Signed-off-by: Hao Lu <haolu@nvidia.com> Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com> Co-authored-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>	2025-04-21 10:01:33 +08:00
Naveassaf	f7c2eb4fa2	Update Nemotron Super and Ultra in Supported Models and add an example (#3632 ) * Update Nemotron Super and Ultra in Supported Models and add an example Signed-off-by: Nave Assaf <nassaf@nvidia.com> * Update README link to match new examples structure Signed-off-by: Nave Assaf <nassaf@nvidia.com> --------- Signed-off-by: Nave Assaf <nassaf@nvidia.com>	2025-04-20 21:14:33 +08:00
QI JUN	d51ae53940	move the reset models into `examples/models/core` directory (#3555 ) * move rest models to examples/models/core directory Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * update multimodal readme Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix example path Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix cpp test Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix tensorrt test Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> --------- Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-04-19 20:48:59 -07:00
brb-nv	c35d2a7532	test: Get Eagle tests working (#3593 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-04-20 00:50:57 +08:00
Yechan Kim	5460d18b10	feat: trtllm-serve multimodal support (#3590 ) * feat: trtllm-serve multimodal support Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * remove disable argument Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * remove disable Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * add and separate tests and move the doc Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * remove block_resue arg from serve.py Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> --------- Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-04-19 05:01:28 +08:00
mayani-nv	ce8329646f	Update run.py for draft_target_model (#3615 ) This change makes the draft target model works without mismatch in the vocab size Signed-off-by: mayani-nv <67936769+mayani-nv@users.noreply.github.com> Co-authored-by: rakib-hasan <rhasan@nvidia.com>	2025-04-19 01:01:50 +08:00
Zheng Duan	bce7ea8c38	test: add kv cache event tests for disagg workers (#3602 )	2025-04-18 18:30:19 +08:00
Dom Brown	dbd9a83b0d	feat: Integrate GPUDirect Storage (GDS) into Executor API (#3582 ) * feat: Integrate GPUDirect Storage (GDS) into Executor API Squash of several dev commits Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>	2025-04-18 15:59:21 +08:00
Zheyu Fu	90a28b917f	feat: Add Dynasor-CoT in scaffolding examples. (#3501 ) Signed-off-by: Zheyu Fu <zheyufu2@gmail.com> Co-authored-by: Junda Chen <32371474+GindaChen@users.noreply.github.com> Co-authored-by: Yichao Fu <57950249+fuyichao2000@users.noreply.github.com> Co-authored-by: Andy Dai <zhongdongmin@nvidia.com>	2025-04-18 07:48:01 +00:00
Yuan Tong	0b0e6d8a0a	refactor: Clean up CMakeLists.txt (#3479 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>	2025-04-18 14:39:29 +08:00
rakib-hasan	ff3b741045	feat: adding multimodal (only image for now) support in trtllm-bench (#3490 ) * feat: adding multimodal (only image for now) support in trtllm-bench Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * fix: add in load_dataset() calls to maintain the v2.19.2 behavior Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * re-adding prompt_token_ids and using that for prompt_len Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * updating the datasets version in examples as well Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * api changes are not needed Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * moving datasets requirement and removing a missed api change Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * addressing review comments Signed-off-by: Rakib Hasan <rhasan@nvidia.com> * refactoring the quickstart example Signed-off-by: Rakib Hasan <rhasan@nvidia.com> --------- Signed-off-by: Rakib Hasan <rhasan@nvidia.com>	2025-04-18 07:06:16 +08:00
Tao Li @ NVIDIA	e4476bf521	update fp8 doc (#3647 ) (#3650 ) Signed-off-by: taoli <litaotju@users.noreply.github.com> Co-authored-by: taoli <litaotju@users.noreply.github.com>	2025-04-17 13:37:08 +08:00
Zhanrui Sun	9d88ee3e45	chore: bump version to 0.20.0rc0 (#3561 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-16 11:41:21 +08:00
narutolhy	ccd73c71a5	feat: Add stream generation task scaffolding examples (#3527 ) * stream generation task/controller Signed-off-by: narutolhy <582909902@qq.com> * edit README Signed-off-by: narutolhy <582909902@qq.com> * rename README Signed-off-by: narutolhy <582909902@qq.com> --------- Signed-off-by: narutolhy <582909902@qq.com>	2025-04-16 11:33:55 +08:00
Kaiyu Xie	f5f68ded26	Minor fixes for documents (#3577 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-04-16 07:47:18 +08:00
Pengyun Lin	1899e71364	doc: add genai-perf benchmark & slurm multi-node for trtllm-serve doc (#3407 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com>	2025-04-16 00:11:58 +08:00
nv-guomingz	39bdb1fe1c	docs:update llm api examples and customizations sections' links. (#3566 ) Signed-off-by: nv-guomingz <37257613+nv-guomingz@users.noreply.github.com>	2025-04-15 13:55:22 +08:00
Bo Li	5eae397b3b	doc: Update instructions to enable FP8 MLA for Deepseek. (#3488 ) * doc: Update doc to enable FP8 MLA for Deepseek. Signed-off-by: Bo Li <bobboli0202@gmail.com> * Update. Signed-off-by: Bo Li <bobboli0202@gmail.com> * Update. Signed-off-by: Bo Li <bobboli0202@gmail.com> * Update the status on Hopper and Blackwell. Signed-off-by: Bo Li <bobboli0202@gmail.com> * Update. Signed-off-by: Bo Li <bobboli0202@gmail.com> * Update table of contents. Signed-off-by: Bo Li <bobboli0202@gmail.com> --------- Signed-off-by: Bo Li <bobboli0202@gmail.com> Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>	2025-04-15 13:12:33 +08:00
Zhanrui Sun	714ff3eedd	chore: bump version to 0.19.0rc0 (#3535 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-14 18:11:20 +08:00
Zhanrui Sun	ee4ce0379d	chore: bump version to 0.19.0rc0 (#3514 ) * chore: bump version to 0.19.0.rc0 Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> * Update README Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> --------- Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-14 17:32:30 +08:00
Kaiyu Xie	f99be2726f	doc: Add example section for multi-node DeepSeek R1 benchmark on GB200 (#3519 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-04-14 16:45:55 +08:00
brb-nv	44090a5388	Add support for Phi-4-MM (#3296 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-04-14 14:24:10 +08:00
Yan Chunwei	b37c5c0a4d	make LLM-API slurm examples executable (#3402 ) Signed-off-by: chunweiy <328693+Superjomn@users.noreply.github.com>	2025-04-13 21:42:45 +08:00
QI JUN	d167cbd5bb	refactor: remove ParallelConfig in tensorrt_llm._torch.distributed module (#3370 ) * remove tensorrt_llm._torch.distributed.ParallelConfig Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * clean Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix embedding test Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix comments Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * polish Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * fix ci Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * rebase Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> --------- Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com>	2025-04-11 15:34:20 -07:00
Shunkangz	ea050084ad	feat: Add support of chat completion in PD (#2985 ) * Add support of chat completion in PD Add support of include_usage in PD Reformat * Remove redundant code Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> * Refactor code Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> * Add chat completion test Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> * Refactor code Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> --------- Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>	2025-04-11 17:53:28 +08:00
amitz-nv	a6a2ae6cc1	chore: Rename nvsmall to nemotron nas (#3447 ) * Rename nvsmall to nemotron NAS * Revert nvsmall to nemotron_nas rename in paths in tests that access llm_models_root/nvsmall/tests * Add NemotronNAS to pytorch supported models table Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>	2025-04-10 23:16:52 +08:00
wm2012011492	af05749e90	feat: add qwen2 moe to torch flow; fix wrong imported KvCacheConfig in gpqa… (#3369 ) * add qwen2 moe to torch flow; fix wrong imported KvCacheConfig in gpqa_llmapi.py Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com> * fix coding style Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com> * add unittest Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com> --------- Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com> Co-authored-by: mengw <12670782+wm2012011492@users.noreply.github.com>	2025-04-10 22:45:57 +08:00
Kefeng-Duan	67949f7c39	Update README and add benchmarking blog for DeepSeek-R1 (#3232 ) - Added a new entry in the README for the published benchmarking best practices for DeepSeek-R1. - Introduced a new blog post detailing performance benchmarking configurations and procedures for DeepSeek-R1 in TensorRT-LLM, including installation, dataset preparation, and benchmarking steps for both B200 and H200 GPUs. Signed-off-by: taoli <litaotju@users.noreply.github.com> Co-authored-by: taoli <litaotju@users.noreply.github.com>	2025-04-10 17:00:49 +08:00
brb-nv	c59abae436	feat: Add Gemma3 text-only model support (#3247 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-04-10 12:34:58 +08:00
Yechan Kim	943218b54a	feat: Add Qwen2.5-VL and refactor Qwen2-VL (#3156 ) * feat: Add Qwen2.5-VL and refactor Qwen2-VL Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix yapf and codespell Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * add test Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix test_e2e Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * generalize get_rope_index Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix qwen2.5-vl on REAME Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix test Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * fix image test Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> --------- Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>	2025-04-10 04:09:03 +08:00
WeiHaocheng	6eee15900e	feat: Enhance the integrated robustness of scaffolding with __init__.py #3305 (#3312 ) Signed-off-by: fredw (generated by with_the_same_user script) <20514172+WeiHaocheng@users.noreply.github.com>	2025-04-09 21:13:47 +08:00
wili	6f1b2cdb83	Doc: update steps of using Draft-Target-Model (DTM) in the documents. (#3366 ) Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com>	2025-04-09 17:35:01 +08:00
Mike Iovine	5bdf997963	Add Llama 4 (#3302 ) Signed-off-by: Mike Iovine <miovine@nvidia.com>	2025-04-09 03:35:21 +08:00
wili	54ad95eaa8	Feat: Variable-Beam-Width-Search (VBWS) part3 (#3338 ) * feat/Variable-Beam-Width-Search-Part3, v1.0 Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com> * feat/Variable-Beam-Width-Search-Part3, v1.1 Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com> * feat/Variable-Beam-Width-Search-Part3, v1.2 Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com> --------- Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com> Co-authored-by: wili-65535 <wili-65535@user.noreply.github.com>	2025-04-08 23:51:27 +08:00
sugunav14	84fc07b011	feat: [TRTLLM-3510] DeepseekV3 support in AutoDeploy (#3281 ) Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>	2025-04-08 21:47:57 +08:00
Zhanrui Sun	63b0194c50	chore: bump version to 0.19.0.dev2025041500 (#3360 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-08 20:45:27 +08:00
yuxianq	7b03350527	Add thread leak check and fix thread/memory leak issues. (#3270 ) Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>	2025-04-08 19:03:18 +08:00
amirkl94	e04f6a1b9b	fix: Fix p-tuning test bug (#3326 ) * fix: Fix p-tuning test bug * A change in the vocab_size calculation for T5Tokenizer, introduced in transformers version 4.34, caused addition of incorrect vtokens for ptuning. In general, instead of adding tokens which are outside the vocabulary, tokens inside the vocabulary were added. Signed-off-by: Amir Klein <203507526+amirkl94@users.noreply.github.com>	2025-04-08 17:14:00 +08:00
Gabriel Wu	f1655afb0d	feat: enable DeepGEMM by default (#3341 ) Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>	2025-04-08 13:58:57 +08:00
Chuang Zhu	1c88af1378	feat: use cudaMalloc to allocate kvCache (#3303 )	2025-04-08 10:59:14 +08:00
Chuang Zhu	f3237e52ed	update readme for disaggregated (#3323 ) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>	2025-04-07 21:29:15 +08:00
Gabriel Wu	376731013d	feat: use NVRTC for DeepGEMM JIT compilation (#3239 ) * feat: use NVRTC for DeepGEMM JIT compilation Signed-off-by: Zihua Wu * fix: add license Signed-off-by: Zihua Wu * feat: store NVRTC JIT results in memory by default Signed-off-by: Zihua Wu * feat: refinement Signed-off-by: Zihua Wu * feat: refinement Signed-off-by: Zihua Wu * test: set timeout to 7200 Signed-off-by: Zihua Wu --------- Signed-off-by: Zihua Wu	2025-04-07 20:29:23 +08:00
tburt-nv	7a659885e3	chore: remove usernames from comments (#3291 ) Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>	2025-04-05 13:44:28 +08:00
Yan Chunwei	b21cfcfed1	chore: refactor the LlmArgs with Pydantic and migrate remaining pybinding configs to python (#3025 ) * make LlmArgs Pydantic Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * amending doc fix api_stability fix tests Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * restore yaml groups refine StackTrace singleton clean tests Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * fix trtllm-bench fix pytorch Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * fix serve distagg Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> * fix Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> --------- Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-04-05 13:31:48 +08:00
Pengyun Lin	f25c7cefb4	doc: refactor trtllm-serve examples and doc (#3187 ) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com> Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-04-04 11:40:43 +08:00
pcastonguay	b763051ba4	chore: Refactor disaggregated serving scripts (#3073 ) * chore: Refactor to reduce duplicated code in disagg server, reuse trtllm-serve Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> * Updating README, removing launch script Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> * Fixing integration tests Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> * Adding scripts to populate urls section of disagg config based on SLURM env vars Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> --------- Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com>	2025-04-03 14:55:05 -04:00
Kaiyu Xie	385a01055c	doc: Add serving section for DS V3 document (#3262 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-04-03 21:57:48 +08:00
Fanrong Li	11624a8e96	fix deepseek-v3 mtp doc. (#3272 ) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: juney-nvidia <143764042+juney-nvidia@users.noreply.github.com>	2025-04-03 21:12:17 +08:00
Yechan Kim	c7533d271f	doc: add supported-models on PyTorch example (#3179 ) * doc: add supported-models on PyTorch example Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> * remove vision support from Llama3.2 Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> --------- Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: juney-nvidia <143764042+juney-nvidia@users.noreply.github.com>	2025-04-03 21:09:25 +08:00
Enwei Zhu	d3948cd9b2	fix: GPT-Next convert failure (#3220 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-04-02 17:14:39 +08:00
WeiHaocheng	e64c565750	doc: add a directory for scaffolding contributors (#3224 ) Signed-off-by: fredw <20514172+WeiHaocheng@users.noreply.github.com>	2025-04-02 16:08:00 +08:00
brb-nv	1fe3e30356	Add support for Phi-4-mini (#2990 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-04-02 08:34:39 +08:00
Zhanrui Sun	42963baacd	chore: bump version to 0.19.0.dev2025040800 (#3171 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-04-02 08:21:55 +08:00
Fridah-nv	a5f32f46fd	fix: [AutoDeploy] Update README.md (#3072 ) * update support matrix and add toggle list Signed-off-by: fridah <201670829+Fridah-nv@users.noreply.github.com> * Update README.md Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> * Update README.md Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com> --------- Signed-off-by: fridah <201670829+Fridah-nv@users.noreply.github.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>	2025-04-01 16:16:36 -07:00
Enwei Zhu	b2f69db507	test: Accuracy test improvement (Part 3.1): Extend accuracy test suite with LLM API and initial implementation of `trtllm-eval` (#3167 ) * add eval_llmapi Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> tmp commit port to CLI tool Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> move Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> setup llmapi Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> fix spec_dec_algo Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> _update_from_hf_quant_config Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> migrate test_pytorch.py Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> fix fp8 block scales Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> fix fp8 rowwise Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> adj alpha Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> move test_pytorch.py cases Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> move Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> rename test_accuracy.py to test_cli.py Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> clean Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix cnn_dailymail Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * renaming to cli flow Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * rename MMLU Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * rename Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * add error Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-04-01 22:20:29 +08:00
WeiHaocheng	ff35af77ea	feat: refactor scaffolding worker and support openai api worker (#3166 ) Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com> Signed-off-by: fredw <20514172+WeiHaocheng@users.noreply.github.com>	2025-04-01 18:31:52 +08:00
brb-nv	727d78e785	Support prequantized fp8 ckpt for nemotron-mini-4b-instruct (#3046 ) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>	2025-04-01 14:52:09 +08:00
Yan Chunwei	7575dd00e7	add slurm script examples for llm-api (#3135 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-04-01 14:31:57 +08:00
Zhanrui Sun	36ac5e78ed	chore: bump version to 0.19.0.dev2025040100 (#3152 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-03-31 16:36:06 +08:00
bhsueh_NV	322ac565fc	chore: clean some ci of qa test (#3083 ) * move some models to examples/models/contrib Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * update the document Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * remove arctic, blip2, cogvlm, dbrx from qa test list Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * remove tests of dit, mmdit and stdit from qa test Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * remove grok, jais, sdxl, skywork, smaug from qa test list Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * re-organize the glm examples Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * fix issues after running pre-commit Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * fix some typo in glm_4_9b readme Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> * fix bug Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com> --------- Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-03-31 14:30:41 +08:00
musvaage	88e1c90fd0	doc: use alert formatting (#3153 ) Signed-off-by: musvaage <musvaage@users.noreply.github.com> Co-authored-by: musvaage <musvaage@users.noreply.github.com>	2025-03-31 07:30:52 +08:00
Mike Iovine	5416966ddb	Add initial EAGLE-3 implementation (#3035 ) Signed-off-by: Mike Iovine <miovine@nvidia.com>	2025-03-29 22:31:24 +08:00
Erin	c75d7cd684	move BuildConfig functional args to llmargs (#3036 ) Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>	2025-03-29 02:20:18 +08:00
Fanrong Li	644a01cbbe	test: Add gpqa tests for DeepSeek models (#3063 ) * Add gpqa accuracy test script * Add gpqa accuracy tests * Update DeepSeek-v3 doc * Update qa test list --------- Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>	2025-03-27 19:47:06 +08:00
Suyog Gupta	047f2b234d	perf: [AutoDeploy] Enable AutoDeploy as a backend in trtllm-bench (#3041 ) * Enable AutoDeploy as a backend in trtllm-bench Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * update how caches are resized Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * fix: files permission from 100755 to 100644 Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * some comments Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * lint Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * lint Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * lint Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * lint Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * Fix function name Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * refactor Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * Remove spurious change Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * Add cursor generated doc strings Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * re-enable ad test Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * some perf cleanup Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * debug ci Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * ensure that overlap scheduler is enabled Signed-off-by: Suyog Gupta <suyogg@nvidia.com> * Reorder the tests Signed-off-by: Suyog Gupta <suyogg@nvidia.com> --------- Signed-off-by: Suyog Gupta <suyogg@nvidia.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-03-26 14:33:14 -07:00
wili	3e035f2219	v1.2 (#3082 ) Signed-off-by: wili <wili@nvidia.com>	2025-03-26 23:31:29 +08:00
Ivy Zhang	3e116c9687	test: add random image test for llama-3.2-11b-vision (#3055 ) * add random image test for llama-3.2-11b-vision Signed-off-by: Ivy Zhang <yanzh@nvidia.com> * rename case Signed-off-by: Ivy Zhang <yanzh@nvidia.com> --------- Signed-off-by: Ivy Zhang <yanzh@nvidia.com> Co-authored-by: Larry <larryx@nvidia.com> CI got Passed: https://nv/trt-llm-cicd/job/helpers/job/PR_Github/522/	2025-03-26 15:38:16 +08:00
Aurelien Chartier	0ec7b5701f	chore: Handle qwen2audio inputs ids expansion during processing (#3080 ) * Handle qwen2audio inputs ids expansion during processing Signed-off-by: Aurelien Chartier <achartier@nvidia.com> * remove more dead code Signed-off-by: Aurelien Chartier <achartier@nvidia.com> * fix yapf Signed-off-by: Aurelien Chartier <achartier@nvidia.com> --------- Signed-off-by: Aurelien Chartier <achartier@nvidia.com> Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-03-26 15:00:27 +08:00
Yechan Kim	3c7cb6629c	Add EXAONE-Deep (#3054 ) Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com> Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>	2025-03-26 14:24:04 +08:00
DylanChen-NV	1ac0566a93	fix: fix for cp > kvHeadNum (#3002 ) * fix for cp > kvHeadNum Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com> * fix for None kv_head_num Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com> --------- Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>	2025-03-26 12:39:02 +08:00
Xiaowei Wang	d9acce72bb	doc: Update DeepSeekV3 doc (#3052 ) * Update DeepGEMM and flashMLA related content * Add single-node command for deepgemm * Fix spelling --------- Signed-off-by: xiaoweiw-nv <100599594+xiaoweiw-nv@users.noreply.github.com>	2025-03-25 18:17:26 +08:00
Zhanrui Sun	c2ffce7dbd	chore: bump version to "0.19.0.dev2025032500" (#3019 ) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>	2025-03-25 10:04:17 +08:00
Yan Chunwei	c29cebf79d	Deprecate model_api examples (#2999 ) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>	2025-03-25 09:37:20 +08:00
Enwei Zhu	705eef68c2	test: Accuracy test improvement (Part 2): Incorporate mmlu to accuracy test suite (#2982 ) * Accuracy test improvement (Part 2) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * WAR OOM Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> update Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> * fix Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> --------- Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-03-25 07:34:10 +08:00
Pradeep Raj Prabhu Raj	5b4a5014d1	Fix: wrong path to constraints.txt in bloom/requirements.txt (#3003 ) Signed-off-by: Pradeep Raj Prabhu Raj <pradeepraj18062002@gmail.com>	2025-03-24 23:03:40 +08:00
QI JUN	832ea997f6	chore: Simplify quickstart of PyTorch flow (#3000 ) * simplify quickstart of PyTorch flow Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> * clean Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> --------- Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>	2025-03-24 14:32:17 +08:00
Michael Gschwind	08b45d1bb9	Update README.md (#2862 ) fix various typos Signed-off-by: Michael Gschwind <61328285+mikekgfb@users.noreply.github.com>	2025-03-24 13:46:09 +08:00
bhsueh_NV	7413cb555a	relax the limitation of setuptools (#2992 ) Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>	2025-03-24 13:36:10 +08:00
Laikh Tewari	456a850e66	Claim support for QwQ 32B (#2877 ) Signed-off-by: Laikh Tewari <laikhtewari1@gmail.com>	2025-03-24 13:05:15 +08:00
Enwei Zhu	c03d59817f	fix: LLM API logits processor example comments (#2962 ) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>	2025-03-24 12:22:12 +08:00
Kaiyu Xie	2631f21089	Update (#2978 ) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2025-03-23 16:39:35 +08:00
Kaiyu Xie	3aa6b11d13	Update TensorRT-LLM (#2936 ) * Update TensorRT-LLM --------- Co-authored-by: changcui <cuichang147@gmail.com>	2025-03-18 21:25:19 +08:00
Kaiyu Xie	9b931c0f63	Update TensorRT-LLM (#2873 )	2025-03-11 21:13:42 +08:00
Kaiyu Xie	225b77667c	Fix .gitmodules (#2852 )	2025-03-04 22:34:09 +08:00
Kaiyu Xie	77d7fe1eb2	Update TensorRT-LLM (#2849 ) * Update TensorRT-LLM --------- Co-authored-by: aotman <chenhangatm@gmail.com>	2025-03-04 18:44:00 +08:00
Kaiyu Xie	ab5b19e027	Update TensorRT-LLM (#2820 )	2025-02-25 21:21:49 +08:00
Kaiyu Xie	2ea17cdad2	Update TensorRT-LLM (#2792 ) * Update TensorRT-LLM --------- Co-authored-by: jlee <jungmoolee@clika.io>	2025-02-18 21:27:39 +08:00
Kaiyu Xie	e88da961c5	Update TensorRT-LLM (#2783 )	2025-02-13 18:40:22 +08:00
Dan Blanaru	16d2467ea8	Update TensorRT-LLM (#2755 ) * Update TensorRT-LLM --------- Co-authored-by: Denis Kayshev <topenkoff@gmail.com> Co-authored-by: akhoroshev <arthoroshev@gmail.com> Co-authored-by: Patrick Reiter Horn <patrick.horn@gmail.com> Update	2025-02-11 03:01:00 +00:00
Kaiyu Xie	be17881062	Update TensorRT-LLM (#2582 )	2024-12-16 21:50:47 -08:00
Kaiyu Xie	aaacc9bd68	Update TensorRT-LLM (#2562 ) * Update TensorRT-LLM --------- Co-authored-by: Starrick Liu <73152103+StarrickLiu@users.noreply.github.com>	2024-12-11 00:31:05 -08:00
石晓伟	548b5b7310	Update TensorRT-LLM (#2532 ) * blossom-ci.yml: run vulnerability scan on blossom * open source efb18c1256f8c9c3d47b7d0c740b83e5d5ebe0ec --------- Co-authored-by: niukuo <6831097+niukuo@users.noreply.github.com> Co-authored-by: pei0033 <59505847+pei0033@users.noreply.github.com> Co-authored-by: Kyungmin Lee <30465912+lkm2835@users.noreply.github.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2024-12-04 21:16:56 +08:00
Kaiyu Xie	385626572d	Update TensorRT-LLM (#2502 ) * Update TensorRT-LLM --------- Co-authored-by: 岑灿 <yunyi.hyy@alibaba-inc.com>	2024-11-26 16:51:34 +08:00
Kaiyu Xie	535c9cc673	Update TensorRT-LLM (#2460 )	2024-11-19 18:30:34 +08:00
Kaiyu Xie	c629546ce4	Update TensorRT-LLM (#2436 )	2024-11-12 15:27:49 +08:00
Kaiyu Xie	b7868dd1bd	Update TensorRT-LLM (#2413 )	2024-11-05 16:27:06 +08:00
Kaiyu Xie	f14d1d433c	Update TensorRT-LLM (#2389 ) * Update TensorRT-LLM --------- Co-authored-by: Alessio Netti <netti.alessio@gmail.com>	2024-10-29 22:24:38 +08:00
Laikh Tewari	3c46c2794e	Specify Llama 3.x information in example (#2343 )	2024-10-25 16:10:57 +08:00
Kaiyu Xie	1730a587d8	Update TensorRT-LLM (#2363 ) * Update TensorRT-LLM --------- Co-authored-by: tonylek <137782967+tonylek@users.noreply.github.com>	2024-10-22 20:27:35 +08:00
Kaiyu Xie	75057cd036	Update TensorRT-LLM (#2333 ) * Update TensorRT-LLM --------- Co-authored-by: Puneesh Khanna <puneesh.khanna@tii.ae> Co-authored-by: Ethan Zhang <26497102+ethnzhng@users.noreply.github.com>	2024-10-15 15:28:40 +08:00
Kaiyu Xie	8681b3a4c0	open source 4dbf696ae9b74a26829d120b67ab8443d70c8e58 (#2297 ) * Update TensorRT-LLM --------- Co-authored-by: Bhuvanesh Sridharan <bhuvanesh.sridharan@sprinklr.com> Co-authored-by: Qingquan Song <ustcsqq@gmail.com>	2024-10-08 12:19:19 +02:00
Dan Blanaru	48686bca3a	open source 7f370deb0090d885d7518c2b146399ba3933c004 (#2273 ) * Update TensorRT-LLM --------- Co-authored-by: Qingquan Song <ustcsqq@gmail.com>	2024-09-30 13:51:19 +02:00
Kaiyu Xie	40274aac39	Bump version to `0.14.0.dev2024092401` (#2258 )	2024-09-26 10:26:16 +08:00
Kaiyu Xie	e153372759	Update TensorRT-LLM (#2253 ) * Update TensorRT-LLM --------- Co-authored-by: Ivan Sorokin <isorokin@nvidia.com> Co-authored-by: lkm2835 <lkm2835@gmail.com>	2024-09-24 17:27:31 +02:00
Kaiyu Xie	a65dba7aaf	Bump version to `0.14.0.dev2024091700` (#2234 )	2024-09-18 08:58:36 +08:00
Kaiyu Xie	fe7dc6ad4e	Update TensorRT-LLM (#2230 ) * Update TensorRT-LLM --------- Co-authored-by: Yi Wang <yi.wang.2005@gmail.com> Co-authored-by: lkm2835 <lkm2835@gmail.com>	2024-09-17 14:39:09 +08:00
Kaiyu Xie	31ac30e928	Update TensorRT-LLM (#2215 ) * Update TensorRT-LLM --------- Co-authored-by: Sherlock Xu <65327072+Sherlock113@users.noreply.github.com>	2024-09-10 18:21:22 +08:00
Kaiyu Xie	78f5c2936b	Update TensorRT-LLM (#2184 )	2024-09-03 12:14:23 +02:00
石晓伟	b8fc6633ba	Update TensorRT-LLM (#2156 ) Co-authored-by: Bruno Magalhaes <bruno.magalhaes@synthesia.io>	2024-08-27 18:20:59 +08:00
石晓伟	32ed92e449	Update TensorRT-LLM Co-authored-by: Rong Zhou <130957722+ReginaZh@users.noreply.github.com> Co-authored-by: Onur Galoglu <33498883+ogaloglu@users.noreply.github.com> Co-authored-by: Fabian Joswig <fjosw@users.noreply.github.com>	2024-08-20 18:55:15 +08:00
Kaiyu Xie	74b324f667	Update TensorRT-LLM (#2110 )	2024-08-13 22:34:33 +08:00
Kaiyu Xie	be9cd719f7	Update TensorRT-LLM (#2094 ) * Update TensorRT-LLM --------- Co-authored-by: akhoroshev <arthoroshev@gmail.com> Co-authored-by: Fabian Joswig <fjosw@users.noreply.github.com> Co-authored-by: Tayef Shah <tayefshah@gmail.com> Co-authored-by: lfz941 <linfanzai941@gmail.com>	2024-08-07 16:44:43 +08:00
Kaiyu Xie	a681853d38	Update TensorRT-LLM (#2053 )	2024-07-30 21:25:01 +08:00
Kaiyu Xie	93293aa46d	open source 315e9f5ccd286e906d4c0d402fefbf2f69a1febe (#2033 )	2024-07-26 16:19:24 +08:00
Kaiyu Xie	5fa9436e17	Update TensorRT-LLM (#2016 )	2024-07-24 19:50:28 +08:00
dongxuy04	5f26e44ead	open source 3706e7395b9b58994412617992727c8ff2d14c9f (#2010 ) Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>	2024-07-24 05:48:06 +08:00
Kaiyu Xie	bca9a33b02	Update TensorRT-LLM (#2008 ) * Update TensorRT-LLM --------- Co-authored-by: Timur Abishev <abishev.timur@gmail.com> Co-authored-by: MahmoudAshraf97 <hassouna97.ma@gmail.com> Co-authored-by: Saeyoon Oh <saeyoon.oh@furiosa.ai> Co-authored-by: hattizai <hattizai@gmail.com>	2024-07-23 23:05:09 +08:00
Kaiyu Xie	2d234357c6	Update TensorRT-LLM (#1954 ) * Update TensorRT-LLM --------- Co-authored-by: Altair-Alpha <62340011+Altair-Alpha@users.noreply.github.com>	2024-07-16 15:30:25 +08:00
Kaiyu Xie	a96cccafcf	Update TensorRT-LLM (#1918 )	2024-07-09 14:42:22 +08:00
Kaiyu Xie	9dbc5b38ba	Update TensorRT-LLM (#1891 ) * Update TensorRT-LLM --------- Co-authored-by: Marks101 <markus.schnoes@gmx.de> Co-authored-by: lkm2835 <lkm2835@gmail.com>	2024-07-04 14:37:19 +08:00
Kaiyu Xie	9691e12bce	Update TensorRT-LLM (#1835 ) * Update TensorRT-LLM --------- Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>	2024-06-25 21:10:30 +08:00
石晓伟	2a115dae84	Update TensorRT-LLM (#1793 ) Co-authored-by: DreamGenX <x@dreamgen.com> Co-authored-by: Ace-RR <78812427+Ace-RR@users.noreply.github.com> Co-authored-by: bprus <39293131+bprus@users.noreply.github.com> Co-authored-by: janpetrov <janpetrov@icloud.com>	2024-06-18 18:18:23 +08:00
Kaiyu Xie	db4edea1e1	Update TensorRT-LLM (#1763 ) * Update TensorRT-LLM --------- Co-authored-by: Kota Tsuyuzaki <bloodeagle40234@gmail.com> Co-authored-by: Pzzzzz <hello-cd.plus@hotmail.com> Co-authored-by: Patrick Reiter Horn <patrick.horn@gmail.com>	2024-06-11 16:59:02 +08:00
Kaiyu Xie	b777bd6475	Update TensorRT-LLM (#1725 ) * Update TensorRT-LLM --------- Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: Tlntin <TlntinDeng01@Gmail.com> Co-authored-by: ZHENG, Zhen <zhengzhen.z@qq.com> Co-authored-by: Pham Van Ngoan <ngoanpham1196@gmail.com> Co-authored-by: Nathan Price <nathan@abridge.com> Co-authored-by: Tushar Goel <tushar.goel.ml@gmail.com> Co-authored-by: Mati <132419219+matichon-vultureprime@users.noreply.github.com>	2024-06-04 20:26:32 +08:00
Kaiyu Xie	f430a4b447	Update TensorRT-LLM (#1688 ) * Update TensorRT-LLM --------- Co-authored-by: IbrahimAmin <ibrahimamin532@gmail.com> Co-authored-by: Fabian Joswig <fjosw@users.noreply.github.com> Co-authored-by: Pzzzzz <hello-cd.plus@hotmail.com> Co-authored-by: CoderHam <hemant@cohere.com> Co-authored-by: Konstantin Lopuhin <kostia.lopuhin@gmail.com>	2024-05-28 20:07:49 +08:00
Kaiyu Xie	5d8ca2faf7	Update TensorRT-LLM (#1639 ) * Update TensorRT-LLM --------- Co-authored-by: vonjackustc <fga@mail.ustc.edu.cn>	2024-05-21 17:51:02 +08:00
Minwoo Lee	b189b61312	Fix mistral v0.1 build instructions (#1373 )	2024-05-20 18:16:02 +08:00
Kaiyu Xie	bf0a5afc92	Update TensorRT-LLM (#1598 ) * Update TensorRT-LLM	2024-05-14 16:43:41 +08:00
Kaiyu Xie	89ba1b1a67	Update TensorRT-LLM (#1554 )	2024-05-07 23:34:28 +08:00
Kaiyu Xie	06c0e9b1ec	Update TensorRT-LLM (#1530 )	2024-04-30 17:19:10 +08:00
Kaiyu Xie	66ef1df492	Update TensorRT-LLM (#1492 ) * Update TensorRT-LLM --------- Co-authored-by: Loki <lokravi@amazon.com>	2024-04-24 14:44:22 +08:00
Kaiyu Xie	71d8d4d3dc	Update TensorRT-LLM (#1455 )	2024-04-16 19:40:08 +08:00
Kaiyu Xie	035b99e0d0	Update TensorRT-LLM (#1427 ) * Update TensorRT-LLM --------- Co-authored-by: meghagarwal <16129366+megha95@users.noreply.github.com>	2024-04-09 17:03:34 +08:00
Kaiyu Xie	118b3d7e7b	Update TensorRT-LLM (#1387 )	2024-04-01 16:39:43 +08:00
石晓伟	850b6fa1e7	Update TensorRT-LLM (#1358 ) Co-authored-by: Kaiyu <26294424+kaiyux@users.noreply.github.com>	2024-03-26 20:47:14 +08:00
Kaiyu Xie	66ca3378c6	Update TensorRT-LLM (#1315 )	2024-03-19 17:36:42 +08:00
Kaiyu Xie	4bb65f216f	Update TensorRT-LLM (#1274 ) * Update TensorRT-LLM --------- Co-authored-by: meghagarwal <16129366+megha95@users.noreply.github.com> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-03-12 18:15:52 +08:00
Kaiyu Xie	728cc0044b	Update TensorRT-LLM (#1233 ) * Update TensorRT-LLM --------- Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-03-05 18:32:53 +08:00
Ashwinkumar J S	b7c309d1c9	Update requirements.txt (#1146 )	2024-02-27 22:09:31 +08:00
Kaiyu Xie	655524dd82	Update TensorRT-LLM (#1168 ) * Update TensorRT-LLM --------- Co-authored-by: Bhuvanesh Sridharan <bhuvan.sridharan@gmail.com> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-02-27 17:37:34 +08:00
Tejaswin Parthasarathy	3c373ebc5b	fix : remove mentions of context plugin (#1128 ) Update examples/gemma/README.md	2024-02-22 09:26:03 +08:00
byshiue_NV	8f4b4df27e	Update README.md (#1126 ) update ammo branch from 0.7.0 to 0.7.3	2024-02-21 21:59:16 +08:00
Kaiyu Xie	eb8f26c7e4	Update TensorRT-LLM (#1122 ) * Update TensorRT-LLM --------- Co-authored-by: Eddie-Wang1120 <wangjinheng1120@163.com> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-02-21 21:30:55 +08:00
Kaiyu Xie	0f041b7b57	Update TensorRT-LLM (#1098 ) * Update TensorRT-LLM * update submodule * Remove unused binaries	2024-02-18 15:48:08 +08:00
Kaiyu Xie	0ab9d17a59	Update TensorRT-LLM (#1055 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-02-06 18:38:07 +08:00
Kaiyu Xie	e06f537e08	Update TensorRT-LLM (#1019 ) * Update TensorRT-LLM --------- Co-authored-by: erenup <ping.nie@pku.edu.cn> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-01-31 21:55:32 +08:00
Kaiyu Xie	b57221b764	Update TensorRT-LLM (#941 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-01-23 23:22:35 +08:00
Kaiyu Xie	c89653021e	Update TensorRT-LLM (20240116) (#891 ) * Update TensorRT-LLM --------- Co-authored-by: Eddie-Wang1120 <81598289+Eddie-Wang1120@users.noreply.github.com> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-01-16 20:03:11 +08:00
Yan Chunwei	12e82e30b0	init (#848 )	2024-01-09 22:48:48 +08:00
Kaiyu Xie	d879430b04	Update TensorRT-LLM (#846 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-01-09 21:03:35 +08:00
Kaiyu Xie	deaae40bd7	Update TensorRT-LLM (#787 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2024-01-02 17:54:32 +08:00
Kaiyu Xie	d37b507f41	Update TensorRT-LLM main branch (#754 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2023-12-27 17:41:24 +08:00
Kaiyu Xie	a75618df24	Update TensorRT-LLM (#667 ) * Update TensorRT-LLM --------- Co-authored-by: 0xymoro <jerrymeng100@gmail.com> Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2023-12-15 22:14:51 +08:00
Kaiyu Xie	f7eca56161	Update TensorRT-LLM (#613 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: zhang-ge-hao <842720660@qq.com>	2023-12-08 17:49:24 +08:00
Kaiyu Xie	71f60f6df0	Update TensorRT-LLM (#524 )	2023-12-01 22:27:51 +08:00
Kaiyu Xie	711a28d9bf	Update TensorRT-LLM (#465 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2023-11-24 22:12:26 +08:00
Kaiyu Xie	6755a3f077	Update TensorRT-LLM (#422 ) * Update TensorRT-LLM --------- Co-authored-by: Tltin <TltinDeng01@gmail.com> Co-authored-by: zhaohb <zhaohbcloud@126.com> Co-authored-by: Bradley Heilbrun <brad@repl.it> Co-authored-by: nqbao11 <nqbao11.01@gmail.com> Co-authored-by: Nikhil Varghese <nikhil@bot-it.ai>	2023-11-18 00:05:54 +08:00
Kaiyu Xie	b2fd493c16	Update TensorRT-LLM (#349 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2023-11-10 22:30:31 +08:00
Kaiyu Xie	f044eb8d94	Update TensorRT-LLM (#302 ) * Update TensorRT-LLM --------- Co-authored-by: wangruohui <12756472+wangruohui@users.noreply.github.com>	2023-11-07 19:51:58 +08:00
Kaiyu Xie	4de32a86ae	Update TensorRT-LLM (#188 ) * Update batch manager * Update src --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: jdemouth-nvidia <11447840+jdemouth-nvidia@users.noreply.github.com>	2023-10-30 16:06:41 +08:00
Kaiyu Xie	d8b408e6dc	Update TensorRT-LLM (#148 ) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>	2023-10-27 12:10:00 +08:00
Kaiyu Xie	75b6210ff4	Kaiyu/update main (#5 ) * Update * Update	2023-10-18 22:38:53 +08:00
Kevin Xie	39d574ae66	Update	2023-10-11 00:42:09 -07:00
Kevin Xie	027cd518e3	Update	2023-10-10 23:22:17 -07:00
Kevin Xie	6e9e318e91	Update code	2023-09-28 09:00:05 -07:00
Kaiyu Xie	23bc5b7c49	Initial commit	2023-09-20 00:29:41 -07:00

... 6 7 8 9 10 ...

587 Commits