Yukun He 2225745782 [TRTLLM-8129][feat] Allreduce tuning and benchmark script revising (#7870) ... Because we have encountered some perf regression due to using a one-shot kernel instead of NCCL on A100/H100, it will be beneficial if we can have a solid benchmarking of allreduce Op and analyze the data collected from it. Implemented new AllreduceOp heuristic: - Added Linear programming-based heuristic implementation. - Added LUT-based heuristic implementation and corresponding code generation script. AllreduceOp minor fixing: - Fixed a minor issue in AllreduceOp, that the strategy can not be overridden when ONESHOT or TWOSHOT is set. - Fixed a minor TWOSHOT kernel perf issue. - Cleaned up Dispatching code in AllReduceOp. This PR will fix the perf gaps reported in: https://nvbugspro.nvidia.com/bug/5517023 For Deepseek-R1, it shows a performance gain of about 3-4% in concurrency levels of 256 and 512. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com>		2025-11-04 16:42:31 +08:00
..
allgatherOp.cpp	[TRTLLM-7349][feat] Adding new orchestrator type -- ray (#7520)	2025-10-04 08:12:24 +08:00
allreduceOp.cpp	[TRTLLM-8129][feat] Allreduce tuning and benchmark script revising (#7870)	2025-11-04 16:42:31 +08:00
alltoallOp.cpp	[TRTLLM-5966][feat] Helix: add alltoall op (#6815)	2025-09-25 07:18:29 -07:00
attentionOp.cpp	[TRTLLM-8541][feat] Add trtllm-gen sparse MLA kernels to support per-Tensor FP8 KV Cache (#8692)	2025-10-31 14:38:31 -07:00
attentionOp.h	[TRTLLM-8541][feat] Add trtllm-gen sparse MLA kernels to support per-Tensor FP8 KV Cache (#8692)	2025-10-31 14:38:31 -07:00
causalConv1dOp.cpp	fix: fix license bug (#5200)	2025-06-13 18:58:15 +08:00
CMakeLists.txt	[TRTLLM-5966][feat] Helix: add full MLA support for Helix (#8104)	2025-11-04 09:06:58 +08:00
convertSpecDecodingMaskToPackedMaskOp.cpp	Update TensorRT-LLM (#2873)	2025-03-11 21:13:42 +08:00
cublasFp4ScaledMM.cpp	[https://nvbugs/5451205][feat] Add cuBLASLt NVFP4 GEMM backend support (#7943)	2025-10-23 15:55:10 +08:00
cublasScaledMM.cpp	[None][feat] GPT-OSS Sm120/Sm121 Support (#7937)	2025-10-06 16:59:06 -04:00
cublasScaledMM.h	Feat/ds r1 min latency opt round3, add router gemm, fused a gemm, PDL (#4560)	2025-06-14 17:36:22 +08:00
cudaNvfp4MM.cpp	[None][feat] Enable nvfp4 cuda core for sm120 (#8620)	2025-10-29 12:39:03 +08:00
cudaScaledMM.cpp	[NVBUG-5304516/5319741]Qwen2.5VL FP8 support (#5029)	2025-07-09 23:16:42 +08:00
customMoeRoutingOp.cpp	[None] [feat] Enable run_post_quant_allgather for MoE TRTLLM backend (#6794)	2025-09-23 08:24:21 +08:00
cutlassScaledMM.cpp	refactoring: port customized kernels with public cutlass version (#5027)	2025-06-13 16:19:31 +08:00
dsv3FusedAGemmOp.cpp	Feat/ds r1 min latency opt round3, add router gemm, fused a gemm, PDL (#4560)	2025-06-14 17:36:22 +08:00
dsv3RouterGemmOp.cpp	Feat/ds r1 min latency opt round3, add router gemm, fused a gemm, PDL (#4560)	2025-06-14 17:36:22 +08:00
dynamicDecodeOp.cpp	[None][feat] Support ignored prompt length for penalties via new sampling config parameter (#8127)	2025-10-27 13:12:31 -04:00
dynamicDecodeOp.h	[None][feat] Support ignored prompt length for penalties via new sampling config parameter (#8127)	2025-10-27 13:12:31 -04:00
finegrained_mixed_dtype_gemm_thop.cpp	W4A8 GEMM (#6005)	2025-07-20 17:34:57 +03:00
finegrained_mixed_dtype_gemm_thop.h	W4A8 GEMM (#6005)	2025-07-20 17:34:57 +03:00
fmhaPackMaskOp.cpp	Update TensorRT-LLM (#2873)	2025-03-11 21:13:42 +08:00
fp4BatchedQuantize.cpp	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00
fp4BlockScaleMoe.cpp	[None][feat] Update TRTLLM MoE MxFP4 cubins; autotune tileN (#8156)	2025-10-23 09:14:18 +08:00
fp4Gemm.cpp	[TRTLLM-4629] [feat] Add support of CUDA13 and sm103 devices (#7568)	2025-09-16 09:56:18 +08:00
fp4GemmTrtllmGen.cpp	[OMNIML-2336][feat] Add NVFP4 x FP8 (#6809)	2025-09-04 09:03:38 -07:00
fp4Op.cpp	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00
fp4Quantize.cpp	[None][perf] Accelerate global scale calculations for deepEP fp4 combine (#7126)	2025-08-27 00:13:13 +08:00
fp4Quantize.h	[None][perf] Accelerate global scale calculations for deepEP fp4 combine (#7126)	2025-08-27 00:13:13 +08:00
fp4xFp8GemmTrtllmGen.cpp	[OMNIML-2336][feat] Add NVFP4 x FP8 (#6809)	2025-09-04 09:03:38 -07:00
fp8BatchedGemmTrtllmGen.cpp	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00
fp8BlockScaleMoe.cpp	[None][feat] Autotuner can iterate through all tactics for test purposes (#8663)	2025-10-30 13:11:25 +01:00
fp8BlockScalingGemm.cpp	[None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP (#8501)	2025-10-27 10:18:19 +08:00
fp8Op.cpp	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00
fp8Op.h	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00
fp8PerTensorScaleMoe.cpp	[None][feat] Update TRTLLM MoE MxFP4 cubins; autotune tileN (#8156)	2025-10-23 09:14:18 +08:00
fp8PerTensorScalingTrtllmGenGemm.cpp	[TRTLLM-4629] [feat] trtllm-gen kernels support sm103 (#7570)	2025-09-07 10:04:10 +08:00
fp8Quantize.cpp	[None][perf] Use fp8 quant kernel in DS3.2 indexer module (#8701)	2025-10-29 12:45:09 +08:00
fp8RowwiseGemm.cpp	[TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow (#5615)	2025-07-07 18:04:57 +08:00
fusedQKNormRopeOp.cpp	[None][feat] Support Yarn on Qwen3 (#6785)	2025-08-17 07:21:29 +08:00
fusedTopkSoftmax.cpp	refactoring: port customized kernels with public cutlass version (#5027)	2025-06-13 16:19:31 +08:00
gatherTreeOp.cpp	[TRTLLM-5171] chore: Remove GptSession/V1 from TRT workflow (#4092)	2025-05-14 23:10:04 +02:00
groupRmsNormOp.cpp	feat: Add heuristic for GroupRMSNorm kernel selection. (#4047)	2025-05-13 08:52:53 +08:00
helixPostProcessOp.cpp	[TRTLLM-5966][feat] Helix: add full MLA support for Helix (#8104)	2025-11-04 09:06:58 +08:00
llama4MinLatency.cpp	Cherry pick feat/llama4 to main (#4739)	2025-05-30 05:28:40 +08:00
logitsBitmaskOp.cpp	[TRTLLM-7027][feat] Fuse d2t to logitsBitmaskKernel and fix a race condition in one-model spec (#7481)	2025-09-04 23:30:14 +08:00
loraOp.cpp	[TRTLLM-7263][fix] Prevent recreation of cublas handles in lora_grouped_gemm every call (#6968)	2025-08-19 15:39:56 +03:00
mambaConv1dOp.cpp	Update TensorRT-LLM (#2873)	2025-03-11 21:13:42 +08:00
mlaPreprocessOp.cpp	[TRTLLM-7192][feat] optimize MLA chunked prefill && support fp8 mla chunked prefill (#7477)	2025-09-15 21:43:49 +08:00
moeAlltoAllMeta.h	[TRTLLM-7318][feat] MnnvlThroughput AlltoAll implementation. (#7499)	2025-10-27 13:23:06 -04:00
moeAlltoAllOp.cpp	[TRTLLM-7318][feat] MnnvlThroughput AlltoAll implementation. (#7499)	2025-10-27 13:23:06 -04:00
moeCommOp.cpp	[None][fix] Rename: slot_count -> invalid_expert_id (#8783)	2025-11-01 21:36:59 +08:00
moeLoadBalanceOp.cpp	[TRTLLM-6743][feat] Optimize and refactor alltoall in WideEP (#6973)	2025-08-24 08:15:29 -04:00
moeOp.cpp	[TRTLLM-7318][feat] MnnvlThroughput AlltoAll implementation. (#7499)	2025-10-27 13:23:06 -04:00
moeUtilOp.cpp	[TRTLLM-7319][perf] Fuse slicing into MoE. (#6728)	2025-08-25 16:52:30 -04:00
mtpOp.cpp	fix: refactor and fix mtp vanilla (#4762)	2025-06-20 05:23:39 +08:00
mxFp4BlockScaleMoe.cpp	[None][feat] Update TRTLLM MoE MxFP4 cubins; autotune tileN (#8156)	2025-10-23 09:14:18 +08:00
mxFp8Quantize.cpp	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00
ncclCommunicatorOp.cpp	feat: [nvbugs/5261055][nvbugs/5170160] non-invasive pipeline parallelism (#4034)	2025-05-16 04:16:53 +08:00
ncclCommunicatorOp.h	feat: [nvbugs/5261055][nvbugs/5170160] non-invasive pipeline parallelism (#4034)	2025-05-16 04:16:53 +08:00
noAuxTcOp.cpp	[TRTLLM-8637][feat] Optimize the routing kernel for DeepseekV3 (MoE CUTLASS backend); Add support for KimiK2 and Qwen-next (MoE TRTLLM backend) (#7761)	2025-10-20 10:08:31 +08:00
parallelDecodeKVCacheUpdateOp.cpp	Update TensorRT-LLM (#2582)	2024-12-16 21:50:47 -08:00
redrafterCurandOp.cpp	[TRTLLM-5171] chore: Remove GptSession/V1 from TRT workflow (#4092)	2025-05-14 23:10:04 +02:00
reducescatterOp.cpp	[TRTLLM-7349][feat] Adding new orchestrator type -- ray (#7520)	2025-10-04 08:12:24 +08:00
relativeAttentionBiasOp.cpp	Update TensorRT-LLM (#2873)	2025-03-11 21:13:42 +08:00
selectiveScanOp.cpp	Update TensorRT-LLM (#2873)	2025-03-11 21:13:42 +08:00
thUtils.cpp	Update TensorRT-LLM (#2849)	2025-03-04 18:44:00 +08:00
thUtils.h	[TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow (#5615)	2025-07-07 18:04:57 +08:00
tinygemm2.cpp	[None][fix] Restrict tinygemm use to certain SMs (#8182)	2025-10-08 17:55:57 -07:00
userbuffersFinalizeOp.cpp	feat: Introduce UB allocator for pytorch flow (#3257)	2025-04-08 18:39:49 +08:00
userbuffersTensor.cpp	feat: Introduce UB allocator for pytorch flow (#3257)	2025-04-08 18:39:49 +08:00
userbuffersTensor.h	feat: Introduce UB allocator for pytorch flow (#3257)	2025-04-08 18:39:49 +08:00
virtualMemoryAllocator.cpp	[TRTLLM-4406][feat] LLM sleep & wakeup Part 1: virtual device memory (#5034)	2025-08-04 13:51:01 +08:00
weightOnlyQuantGemm.cpp	[TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow (#5850)	2025-07-21 15:17:35 +08:00
weightOnlyQuantGemm.h	[TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow (#5850)	2025-07-21 15:17:35 +08:00
weightOnlyQuantOp.cpp	[None] [feat] Add model gpt-oss (#6645)	2025-08-07 03:04:18 -04:00