kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
af2450c266
TensorRT-LLMs/cpp/tensorrt_llm/kernels
History
Yukun He fd4311e6a3
[TRTLLM-8129][feat] Allreduce tuning and benchmark script revising (#7870) 
Because we have encountered some perf regression due to using a one-shot kernel instead of NCCL on A100/H100, it will be beneficial if we can have a solid benchmarking of allreduce Op and analyze the data collected from it.

Implemented new AllreduceOp heuristic:
- Added Linear programming-based heuristic implementation.
- Added LUT-based heuristic implementation and corresponding code generation script.

AllreduceOp minor fixing:
- Fixed a minor issue in AllreduceOp, that the strategy can not be overridden when ONESHOT or TWOSHOT is set.
- Fixed a minor TWOSHOT kernel perf issue.
- Cleaned up Dispatching code in AllReduceOp.

This PR will fix the perf gaps reported in:
https://nvbugspro.nvidia.com/bug/5517023

For Deepseek-R1, it shows a performance gain of about 3-4% in concurrency levels of 256 and 512.

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
2025-10-16 14:15:25 +08:00
..
beamSearchKernels Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
causalConv1d fix: fix license bug (#5200) 2025-06-13 18:58:15 +08:00
communicationKernels [TRTLLM-8129][feat] Allreduce tuning and benchmark script revising (#7870) 2025-10-16 14:15:25 +08:00
contextFusedMultiHeadAttention [None][chore] remove cubins for ci cases (#7902) 2025-09-24 14:56:31 +08:00
cutlass_kernels [None][chroe] Rename TensorRT-LLM to TensorRT LLM for source code. (#7851) 2025-09-25 21:02:35 +08:00
decoderMaskedMultiheadAttention [None][feat] support JIT mha.cu for SPEC_DEC in runtime (#6078) 2025-09-23 14:56:17 -07:00
dsv3MinLatencyKernels [https://nvbugs/5545522][fix] move PREEXIT in UB kernels to fix accuracy issue (#8318) 2025-10-16 09:50:43 +08:00
flashMLA feat: reduce unnecessary kernel generation (#5476) 2025-07-04 14:37:49 +08:00
fusedLayernormKernels [None] [feat] Add model gpt-oss (#6645) 2025-08-07 03:04:18 -04:00
groupRmsNormKernels feat: Add heuristic for GroupRMSNorm kernel selection. (#4047) 2025-05-13 08:52:53 +08:00
internal_cutlass_kernels [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
llama4MinLatencyKernels feat: reduce unnecessary kernel generation (#5476) 2025-07-04 14:37:49 +08:00
lora chore: Stabilize ABI boundary for internal kernel library (#3117) 2025-04-11 15:07:50 +08:00
moeLoadBalance [None][fix] Fix possible hang issue in WideEP and move some tests to pre-merge (#7262) 2025-08-27 01:39:24 -04:00
selectiveScan fix: fix license bug (#5200) 2025-06-13 18:58:15 +08:00
speculativeDecoding [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
trtllmGenKernels [https://nvbugs/5541494] [fix] Fix missing sm100f/103a kernels and add tests (#8098) 2025-10-07 08:27:55 +08:00
unfusedAttentionKernels [None][feat] Support NVFP4 KV Cache (#6244) 2025-09-01 09:24:52 +08:00
userbuffers [https://nvbugs/5545522][fix] move PREEXIT in UB kernels to fix accuracy issue (#8318) 2025-10-16 09:50:43 +08:00
weightOnlyBatchedGemv [NVBUG-5304516/5319741]Qwen2.5VL FP8 support (#5029) 2025-07-09 23:16:42 +08:00
attentionMask.cu Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
attentionMask.h
banBadWords.cu Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
banBadWords.h
banRepeatNgram.cu Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
banRepeatNgram.h
beamSearchKernels.cu [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
beamSearchKernels.h [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
buildRelativeAttentionBiasKernel.cu
buildRelativeAttentionBiasKernel.h
CMakeLists.txt feat: reduce unnecessary kernel generation (#5476) 2025-07-04 14:37:49 +08:00
cumsumLastDim.cu
cumsumLastDim.h
customAllReduceKernels.cu Cherry pick feat/llama4 to main (#4739) 2025-05-30 05:28:40 +08:00
customAllReduceKernels.h [TRTLLM-8129][feat] Allreduce tuning and benchmark script revising (#7870) 2025-10-16 14:15:25 +08:00
customMoeRoutingKernels.cu [None] [feat] Enable run_post_quant_allgather for MoE TRTLLM backend (#6794) 2025-09-23 08:24:21 +08:00
customMoeRoutingKernels.h [None] [feat] Enable run_post_quant_allgather for MoE TRTLLM backend (#6794) 2025-09-23 08:24:21 +08:00
decoderMaskedMultiheadAttention.cu Update TensorRT-LLM (#2502) 2024-11-26 16:51:34 +08:00
decoderMaskedMultiheadAttention.h [None] [feat] Add model gpt-oss (#6645) 2025-08-07 03:04:18 -04:00
decoderMaskedMultiheadAttentionUtils.h [None] [feat] Add model gpt-oss (#6645) 2025-08-07 03:04:18 -04:00
decodingCommon.cu Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
decodingKernels.cu Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
decodingKernels.h refactor: Improve decoder finalize function (#3077) 2025-03-28 14:33:59 +08:00
delayStream.cu Update (#2978) 2025-03-23 16:39:35 +08:00
delayStream.h Update (#2978) 2025-03-23 16:39:35 +08:00
doraScaling.cu Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
doraScaling.h Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
fmhaDispatcher.cpp [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
fmhaDispatcher.h Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
fusedMoeCommKernels.cu [TRTLLM-6876][feat] Add low precision all2all for mnnvl (#7155) 2025-08-28 18:26:16 +08:00
fusedMoeCommKernels.h [TRTLLM-6876][feat] Add low precision all2all for mnnvl (#7155) 2025-08-28 18:26:16 +08:00
fusedQKNormRopeKernel.cu [None][feat] Support Yarn on Qwen3 (#6785) 2025-08-17 07:21:29 +08:00
fusedQKNormRopeKernel.h [None][feat] Support Yarn on Qwen3 (#6785) 2025-08-17 07:21:29 +08:00
gptKernels.cu [None][feat] Support NVFP4 KV Cache (#6244) 2025-09-01 09:24:52 +08:00
gptKernels.h [None][feat] Support NVFP4 KV Cache (#6244) 2025-09-01 09:24:52 +08:00
groupGemm.cu Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
groupGemm.h Update TensorRT-LLM (#2562) 2024-12-11 00:31:05 -08:00
kvCachePartialCopy.cu [fix] Fix illegal mem access and possible accuracy lose. Cherry-pick … (#5017) 2025-06-09 17:50:57 +08:00
kvCacheUtils.h chore: Improve documentation of Kv_block_array (#5765) 2025-07-05 22:25:27 +02:00
layernormKernels.cu feat: Add support for fp8 rowwise quantization (#4876) 2025-06-14 06:37:48 -07:00
layernormKernels.h feat: Add support for fp8 rowwise quantization (#4876) 2025-06-14 06:37:48 -07:00
logitsBitmask.cu [TRTLLM-7027][feat] Fuse d2t to logitsBitmaskKernel and fix a race condition in one-model spec (#7481) 2025-09-04 23:30:14 +08:00
logitsBitmask.h [TRTLLM-7027][feat] Fuse d2t to logitsBitmaskKernel and fix a race condition in one-model spec (#7481) 2025-09-04 23:30:14 +08:00
lookupKernels.cu
lookupKernels.h
lruKernel.cu
lruKernel.h
mambaConv1dKernels.cu feat: Add FP8 support for SM 120 (#3248) 2025-04-14 16:05:41 -07:00
mambaConv1dKernels.h
mlaChunkedPrefill.cu [TRTLLM-7192][feat] optimize MLA chunked prefill && support fp8 mla chunked prefill (#7477) 2025-09-15 21:43:49 +08:00
mlaChunkedPrefill.cuh [TRTLLM-7192][feat] optimize MLA chunked prefill && support fp8 mla chunked prefill (#7477) 2025-09-15 21:43:49 +08:00
mlaKernels.cu [TRTLLM-5966][feat] Helix: add custom position ids to MLA kernels (#6904) 2025-09-19 20:55:32 +08:00
mlaKernels.h [TRTLLM-5966][feat] Helix: add custom position ids to MLA kernels (#6904) 2025-09-19 20:55:32 +08:00
moeCommKernelsCommon.h [TRTLLM-6743][feat] Optimize and refactor alltoall in WideEP (#6973) 2025-08-24 08:15:29 -04:00
moePrepareKernels.cu [None][fix] Fix possible hang issue in WideEP and move some tests to pre-merge (#7262) 2025-08-27 01:39:24 -04:00
moePrepareKernels.h [TRTLLM-6743][feat] Optimize and refactor alltoall in WideEP (#6973) 2025-08-24 08:15:29 -04:00
moeTopKFuncs.cuh [https://nvbugs/5392414] [fix] Add customized default routing method (#6818) 2025-08-21 16:58:41 +08:00
multiHeadAttentionCommon.h [TRTLLM-4629] [feat] trtllm-gen kernels support sm103 (#7570) 2025-09-07 10:04:10 +08:00
noAuxTcKernels.cu Update (#2978) 2025-03-23 16:39:35 +08:00
noAuxTcKernels.h Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
penaltyKernels.cu Update TensorRT-LLM (#2849) 2025-03-04 18:44:00 +08:00
penaltyKernels.h Update TensorRT-LLM (#2502) 2024-11-26 16:51:34 +08:00
penaltyTypes.h
preQuantScaleKernel.cu chore: Mass integration of release/0.20. (#4871) 2025-06-04 14:12:27 +08:00
preQuantScaleKernel.h chore: Mass integration of release/0.20. (#4871) 2025-06-04 14:12:27 +08:00
qserveGemm.h Update TensorRT-LLM (#2436) 2024-11-12 15:27:49 +08:00
qserveGemmPerChannel.cu Update TensorRT-LLM (#2532) 2024-12-04 21:16:56 +08:00
qserveGemmPerGroup.cu Update TensorRT-LLM (#2502) 2024-11-26 16:51:34 +08:00
quantization.cu [None][perf] Accelerate global scale calculations for deepEP fp4 combine (#7126) 2025-08-27 00:13:13 +08:00
quantization.cuh [None] [feat] Add model gpt-oss (#6645) 2025-08-07 03:04:18 -04:00
quantization.h [None][perf] Accelerate global scale calculations for deepEP fp4 combine (#7126) 2025-08-27 00:13:13 +08:00
recoverFromRingAtten.cu [None][feat] Add support for Hopper MLA chunked prefill (#6655) 2025-08-14 10:39:26 +08:00
recoverFromRingAtten.h Support RingAttention in the BertAttention plugin and the DiT model (#3661) 2025-05-09 08:06:54 +08:00
rmsnormKernels.cu Update TensorRT-LLM (#2436) 2024-11-12 15:27:49 +08:00
rmsnormKernels.h Update TensorRT-LLM (#2436) 2024-11-12 15:27:49 +08:00
sageAttentionKernels.cu [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
sageAttentionKernels.h Update TensorRT-LLM (#2849) 2025-03-04 18:44:00 +08:00
samplingAirTopPKernels.cu Update TensorRT-LLM (#2783) 2025-02-13 18:40:22 +08:00
samplingTopKKernels.cu Update TensorRT-LLM (#2849) 2025-03-04 18:44:00 +08:00
samplingTopKKernels.h [TRTLLM-6785][feat] BREAKING CHANGE Enable TRTLLM sampler by default (#6216) 2025-08-07 22:19:37 -04:00
samplingTopPKernels.cu chore: remove usernames from comments (#3291) 2025-04-05 13:44:28 +08:00
samplingTopPKernels.h Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
splitkGroupGemm.cu Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
splitkGroupGemm.h Update TensorRT-LLM (#2792) 2025-02-18 21:27:39 +08:00
stopCriteriaKernels.cu Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
stopCriteriaKernels.h
topkLastDim.cu [None][chore] Mass integration of release/1.0 (#6864) 2025-08-22 09:25:15 +08:00
topkLastDim.h Update TensorRT-LLM (#2436) 2024-11-12 15:27:49 +08:00
unfusedAttentionKernels.cu fix: fix for cp > kvHeadNum (#3002) 2025-03-26 12:39:02 +08:00
unfusedAttentionKernels.h [None][feat] Support NVFP4 KV Cache (#6244) 2025-09-01 09:24:52 +08:00
xqaDispatcher.cpp [TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568) 2025-09-16 09:56:18 +08:00
xqaDispatcher.h [None][feat] Support NVFP4 KV Cache (#6244) 2025-09-01 09:24:52 +08:00