kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
539 Commits 25 Branches 59 Tags 2.1 GiB
release/0.19
Commit Graph

539 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yukun He
ff82aef99b
Fix the issues related to fused moe path. (#3435) 
* One of the tactic is not supported during dispatch.
* final_hidden_states should be unpacked if it is not min_latency_mode.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
 2025-04-11 21:41:15 +08:00
liji-nv
b168adba70
feat: Add NVFP4 UB pattern optimization pass in torch compile (#3371) 
* feat: Add NVFP4 UB pattern optimization pass in torch compile

* Add an additional flag for UB fp4 pattern to avoid inverse the scale
* Add NVFP4 related UB patterns

Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>

* Update atol, some points fails for B200 umbriel.

Signed-off-by: liji-nv <59594262+liji-nv@users.noreply.github.com>

---------

Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
Signed-off-by: liji-nv <59594262+liji-nv@users.noreply.github.com>
 2025-04-11 21:25:29 +08:00
Shunkangz
ea050084ad
feat: Add support of chat completion in PD (#2985) 
* Add support of chat completion in PD

Add support of include_usage in PD


Reformat


* Remove redundant code

Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>

* Refactor code

Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>

* Add chat completion test

Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>

* Refactor code

Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>

---------

Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
 2025-04-11 17:53:28 +08:00
Yechan Kim
5bc6f093c8
fix: mllama e2e pytorch flow fix (#3397) 
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
 2025-04-11 17:33:15 +08:00
Ivy Zhang
20e54e5c89
test: add cuda visible device constraint for phi_1gpu test (#3364) 
Signed-off-by: Ivy Zhang <yanzh@nvidia.com>
 2025-04-11 17:14:52 +08:00
Ivy Zhang
d998832b33
test: add torch flow test case in qa test list (#3404) 
Signed-off-by: Ivy Zhang <yanzh@nvidia.com>
 2025-04-11 16:57:41 +08:00
pansicheng
143edc8153
fix partialMatch (#3413) 
Signed-off-by: pansicheng <sicheng.pan.chn@gmail.com>
 2025-04-11 16:42:52 +08:00
Yiqing Yan
0d351317c2
Waive failure post-merge tests (#3472) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-11 16:23:07 +08:00
Yuan Tong
a139eae425
chore: Stabilize ABI boundary for internal kernel library (#3117) 
chore: Stabilize ABI boundary for internal kernel library

Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>
 2025-04-11 15:07:50 +08:00
Enwei Zhu
410f56357e
test: Waive torch compile tests (#3471) 
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
 2025-04-11 13:38:05 +08:00
QI JUN
16ca45747b
always trigger multi gpu test to protect modeling_llama.py and modeling_deepseekv3.py (#3434) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-11 13:19:23 +08:00
wili
5142c783c0
fix: Beam Search Diversity (#3375) 
Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com>
Co-authored-by: wili-65535 <wili-65535@user.noreply.github.com>
 2025-04-11 11:58:59 +08:00
QI JUN
1e2a339642
waive unittest/_torch/multi_gpu (#3464) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-11 09:59:16 +08:00
QI JUN
6cef10068a
waive a test case of llama 3.1 with torch compile (#3461) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-11 09:15:19 +08:00
tburt-nv
5616c0d232
add precommit check to github actions (#3129) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
 2025-04-11 06:40:53 +08:00
Dom Brown
a8310b01dc
feat: trtllm-gen fp4 GEMM for pytorch workflow (#3423) 
* feat: trtllm-gen fp4 GEMM

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>

* Clean up

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>

* Remove incorrect header

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>

* Reviewer comment

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>

---------

Signed-off-by: Dom Brown <3886319+DomBrown@users.noreply.github.com>
 2025-04-11 02:28:07 +08:00
Iman Tabrizian
d7f45e50c6
test: disable attention DP tests for single GPU (#3395) 
Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>
 2025-04-11 01:38:17 +08:00
Zhihan Jiang
8300218d21
feat: support llama4 nope layers; support FP8 checkpoint loading; (#3382) 
* Enable NOPE, Fix a rotary embedding bug for gptj_stype_rope, Address PR comment, Properly skip the rotary_embdding for Llama4 ROPE layers

* Add support for FP8 checkpoint, Fix ckpt weighting loading for FP8

* Temporarily disable min_latency_mode for llama4

---------

Co-authored-by: Yilin Fan <yilinf@nvidia.com>
Co-authored-by: Sharan Chetlur <116769508+schetlur-nv@users.noreply.github.com>
 2025-04-10 10:16:42 -07:00
amitz-nv
a6a2ae6cc1
chore: Rename nvsmall to nemotron nas (#3447) 
* Rename nvsmall to nemotron NAS

* Revert nvsmall to nemotron_nas rename in paths in tests that access llm_models_root/nvsmall/tests

* Add NemotronNAS to pytorch supported models table

Signed-off-by: Amit Zuker <203509407+amitz-nv@users.noreply.github.com>
 2025-04-10 23:16:52 +08:00
wm2012011492
af05749e90
feat: add qwen2 moe to torch flow; fix wrong imported KvCacheConfig in gpqa… (#3369) 
* add qwen2 moe to torch flow; fix wrong imported KvCacheConfig in gpqa_llmapi.py

Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com>

* fix coding style

Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com>

* add unittest

Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com>

---------

Signed-off-by: mengw <12670782+wm2012011492@users.noreply.github.com>
Co-authored-by: mengw <12670782+wm2012011492@users.noreply.github.com>
 2025-04-10 22:45:57 +08:00
QI JUN
f5281fffaa
waive some test cases of test_llm_multi_gpu.py (#3452) 
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-10 22:02:35 +08:00
Yan Chunwei
c5e803ba48
chore: code cleanup for error logging and SharedMemory in proxy.py (#3432) 
* cleanup log

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* remove shared-memory

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* remove ExecutorResponse

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

* add assert for postproc

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>

---------

Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com>
 2025-04-10 21:57:06 +08:00
Julien Debache
d7a0bf934c
fix: updating ucxx, which appears to avoid occasional segfaults when profiling (#3420) 
Signed-off-by: jdebache <jdebache@nvidia.com>
 2025-04-10 19:48:20 +08:00
HuiGao-NV
3ade9375ba
feat: Run PyExecutor's inference flow to estimate max_num_tokens for kv_cache_manager (#3092) 
Signed-off-by: Hui Gao <huig@nvidia.com>
 2025-04-10 18:29:40 +08:00
Yiqing Yan
10d2d16247
Waive L0 test (#3442) 
Signed-off-by: Yiqing Yan <yiqingy@nvidia.com>
 2025-04-10 17:43:45 +08:00
Emma Qiao
5023e0d0f4
infra: Update some test description which is out of date (#3437) 
* Update some description which is out of date

Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>

* Apply suggestions from code review

Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>

---------

Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-04-10 17:29:30 +08:00
Kefeng-Duan
67949f7c39
Update README and add benchmarking blog for DeepSeek-R1 (#3232) 
- Added a new entry in the README for the published benchmarking best practices for DeepSeek-R1.
- Introduced a new blog post detailing performance benchmarking configurations and procedures for DeepSeek-R1 in TensorRT-LLM, including installation, dataset preparation, and benchmarking steps for both B200 and H200 GPUs.

Signed-off-by: taoli <litaotju@users.noreply.github.com>
Co-authored-by: taoli <litaotju@users.noreply.github.com>
 2025-04-10 17:00:49 +08:00
bhsueh_NV
cec65bd09a
clean the waive.txt (#3441) 
Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
 2025-04-10 16:20:08 +08:00
xinhe-nv
863d023fd0
test: fix memory leak of tests (#3392) 
Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com>
 2025-04-10 14:31:40 +08:00
tburt-nv
b331d62f98
add sqlite to rocky container (#3114) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
Co-authored-by: Yanchao Lu <yanchaol@nvidia.com>
 2025-04-10 13:30:24 +08:00
yuxianq
16c8f39fc5
feat: Support TLLM_OVERRIDE_LAYER_NUM and TLLM_TRACE_MODEL_FORWARD for debugging (#3417) 
Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
 2025-04-10 13:18:30 +08:00
hlu1
fbcf954d9c
[MLA] Deallocate tensors after use (#3286) 
Signed-off-by: Hao Lu <haolu@nvidia.com>
 2025-04-09 21:36:07 -07:00
brb-nv
c59abae436
feat: Add Gemma3 text-only model support (#3247) 
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
 2025-04-10 12:34:58 +08:00
QI JUN
b5473f7eca
waive llama3.1 8B test cases with pipeline parallelism (#3433) 
* waive llama3.1 8B test cases with pipeline parallelism

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* update

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

---------

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-10 11:07:58 +08:00
Frank
9307ff95ae
fix: Add nested aliases for Llama 4 (#3381) 
* Add nested aliases for Llama 4

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

* Fix missed alias.

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>

---------

Signed-off-by: Frank Di Natale <3429989+FrankD412@users.noreply.github.com>
Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com>
 2025-04-10 10:18:53 +08:00
peaceh-nv
215fb20567
chore : split GptExecutor tests out of gpt tests to reduce single test time (#3412) 
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-10 09:08:15 +08:00
tburt-nv
8d164f40d7
update allowlist (#3428) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
 2025-04-10 06:41:40 +08:00
Yechan Kim
943218b54a
feat: Add Qwen2.5-VL and refactor Qwen2-VL (#3156) 
* feat: Add Qwen2.5-VL and refactor Qwen2-VL

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* fix yapf and codespell

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* add test

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* fix test_e2e

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* generalize get_rope_index

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* fix qwen2.5-vl on REAME

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* fix test

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

* fix image test

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>

---------

Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Co-authored-by: Haohang Huang <31998628+symphonylyh@users.noreply.github.com>
 2025-04-10 04:09:03 +08:00
Maximiliano Levi
996696203f
fix: #3137 speculative decoding and multimodal input support (#3276) 
* fix: broadcast embeddings input when using speculative decoding

Signed-off-by: Maximiliano Levi <maxilevi77@gmail.com>

* fix: use shape tensor instead of tuple

Signed-off-by: Maximiliano Levi <maxilevi77@gmail.com>

* fix: comment

Signed-off-by: Maximiliano Levi <maxilevi77@gmail.com>

---------

Signed-off-by: Maximiliano Levi <maxilevi77@gmail.com>
Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
 2025-04-09 23:40:19 +08:00
danielafrimi
47f5cf6c0d
lora_tests (#3201) 
LoRA tests and layers

Signed-off-by: Ubuntu <dafrimi@nvidia.com>
Co-authored-by: Ubuntu <dafrimi@nvidia.com>
 2025-04-09 18:06:52 +03:00
WeiHaocheng
6eee15900e
feat: Enhance the integrated robustness of scaffolding with __init__.py #3305 (#3312) 
Signed-off-by: fredw (generated by with_the_same_user script) <20514172+WeiHaocheng@users.noreply.github.com>
 2025-04-09 21:13:47 +08:00
石晓伟
c069abc7d8
Update gh pages build script (#3405) 
Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>
 2025-04-09 19:58:38 +08:00
Gabriel Wu
4d78f51608
fix: remove DeepGEMM line info (#3411) 
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
 2025-04-09 18:01:02 +08:00
wili
6f1b2cdb83
Doc: update steps of using Draft-Target-Model (DTM) in the documents. (#3366) 
Signed-off-by: wili-65535 <wili-65535@user.noreply.github.com>
 2025-04-09 17:35:01 +08:00
QI JUN
d0671494cd
chore: fix wheel version <= 0.45.1 (#3391) 
* fix wheel version to 0.45.1

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

* relax version

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

---------

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
 2025-04-09 12:31:55 +08:00
sugunav14
64abb01a36
Fix failing DSV3 unit tests (#3385) 
* Skipping DSV3 module patch unit tests

Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>

* update tested

Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>

* Fixed failing unit test

Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>

---------

Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>
 2025-04-09 11:57:05 +08:00
tburt-nv
3a8443f1e1
extend allowlist (#3379) 
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
 2025-04-09 11:10:42 +08:00
Iman Tabrizian
8401722245
test: Add single gpu disaggregated tests (#3295) 
* test: Add single gpu disaggregated tests

Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>

* Add deepseek with overlap tests

Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>

* Use updated prompt

Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>

* Move test to disaggregated folder

Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>

---------

Signed-off-by: Iman Tabrizian <itabrizian@nvidia.com>
 2025-04-09 09:34:45 +08:00
Tracin
2a2b7bfc66
Fix miss bias add for FP4Linear. (#3361) 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com>
 2025-04-09 09:17:54 +08:00
Mike Iovine
5bdf997963
Add Llama 4 (#3302) 
Signed-off-by: Mike Iovine <miovine@nvidia.com>
 2025-04-09 03:35:21 +08:00