mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-14 06:27:45 +08:00
515dd0d78f
* fp8 kv + bf16 ctx MLA + fp8 gen MLA
Use BF16 for context MLA.
mFP8GenerationMLA and mFP8ContextFMHA shouldn't be enabled together.
Allow mSM==90 for mFP8GenerationMLA==true.
For FMHA, dataTypeKv should be FP8.
For FP8 MLA generation, the output is still in BF16.
Refine debug info for FMHA kernel metadata.
Use inputType, outputType, SM together to hash kernel list.
Add FP8 MLA generation FMHA kernel.
Special WAR of NUM_COMPUTE_GROUPS for MLA generation kernel.
Separate the implementation of fused_multihead_attention_v2.h to CPP and print some debug info if checkIfKernelExist fails.
Refine debug info in fused_multihead_attention_v2.cpp
Correct FP8 MLA metadata.
New kernel provided by Yuxin, which outputs BF16.
smem size is not set correctly, which will lead to illegal mem access.
Yuxin fixed the error in FMHA MLA kernel: previously the BF16 isn't correctly written: some parts are repeatedly written, while some others are untouched.
There are two bmm1 scales that should be set correctly.
New kernel generated by Yuxin.
Modificatiosn to common/attentionOp for FP8 MLA on Hopper using FMHA.
Not necessary. If mFP8GenerationMLA, is_fp8_out is false, so mFP8ContextFMHA is false.
Skip a check in fmhaDispatcher.
Modifications in fmhaRunner:
- Debug dump.
- if (!isFP8GenerationMLA) skips a lot of flag setting.
- TMA descriptor modification for qo (by Yuxin).
Cleanup debug output.
Clean up o tma descriptor modifications.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Resolve conflicts.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Apply the patch of FP8 FlashMLA and resolve conflicts.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Fix compilation error.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Fix compile error.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* pick blackwell support
Signed-off-by: Dylan Chen <191843203+DylanChen-NV@users.noreply.github.com>
* Add copyright notice to fused_multihead_attention_v2.cpp.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Add license.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Add missing license.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Exclude building flashMLA kernels under sm90.
Signed-off-by: Bo Li <bobboli0202@gmail.com>
* Revert "Exclude building flashMLA kernels under sm90."
This reverts commit
|
||
|---|---|---|
| .. | ||
| beamSearchKernels | ||
| contextFusedMultiHeadAttention | ||
| cutlass_kernels | ||
| decoderMaskedMultiheadAttention | ||
| flashMLA | ||
| fusedLayernormKernels | ||
| internal_cutlass_kernels | ||
| lora | ||
| selectiveScan | ||
| speculativeDecoding | ||
| trtllmGenKernels | ||
| unfusedAttentionKernels | ||
| userbuffers | ||
| weightOnlyBatchedGemv | ||
| allReduceFusionKernels.cu | ||
| allReduceFusionKernels.h | ||
| attentionMask.cu | ||
| attentionMask.h | ||
| banBadWords.cu | ||
| banBadWords.h | ||
| banRepeatNgram.cu | ||
| banRepeatNgram.h | ||
| beamSearchKernels.cu | ||
| beamSearchKernels.h | ||
| buildRelativeAttentionBiasKernel.cu | ||
| buildRelativeAttentionBiasKernel.h | ||
| CMakeLists.txt | ||
| cumsumLastDim.cu | ||
| cumsumLastDim.h | ||
| customAllReduceKernels.cu | ||
| customAllReduceKernels.h | ||
| decoderMaskedMultiheadAttention.cu | ||
| decoderMaskedMultiheadAttention.h | ||
| decoderMaskedMultiheadAttentionUtils.h | ||
| decodingCommon.cu | ||
| decodingKernels.cu | ||
| decodingKernels.h | ||
| delayStream.cu | ||
| delayStream.h | ||
| doraScaling.cu | ||
| doraScaling.h | ||
| fmhaDispatcher.cpp | ||
| fmhaDispatcher.h | ||
| gptKernels.cu | ||
| gptKernels.h | ||
| groupGemm.cu | ||
| groupGemm.h | ||
| kvCachePartialCopy.cu | ||
| kvCacheUtils.h | ||
| layernormKernels.cu | ||
| layernormKernels.h | ||
| logitsBitmask.cu | ||
| logitsBitmask.h | ||
| lookupKernels.cu | ||
| lookupKernels.h | ||
| lruKernel.cu | ||
| lruKernel.h | ||
| mambaConv1dKernels.cu | ||
| mambaConv1dKernels.h | ||
| mlaKernels.cu | ||
| mlaKernels.h | ||
| multiHeadAttentionCommon.h | ||
| noAuxTcKernels.cu | ||
| noAuxTcKernels.h | ||
| penaltyKernels.cu | ||
| penaltyKernels.h | ||
| penaltyTypes.h | ||
| preQuantScaleKernel.cu | ||
| preQuantScaleKernel.h | ||
| qserveGemm.h | ||
| qserveGemmPerChannel.cu | ||
| qserveGemmPerGroup.cu | ||
| quantization.cu | ||
| quantization.cuh | ||
| quantization.h | ||
| rmsnormKernels.cu | ||
| rmsnormKernels.h | ||
| sageAttentionKernels.cu | ||
| sageAttentionKernels.h | ||
| samplingAirTopPKernels.cu | ||
| samplingTopKKernels.cu | ||
| samplingTopKKernels.h | ||
| samplingTopPKernels.cu | ||
| samplingTopPKernels.h | ||
| splitkGroupGemm.cu | ||
| splitkGroupGemm.h | ||
| stopCriteriaKernels.cu | ||
| stopCriteriaKernels.h | ||
| topkLastDim.cu | ||
| topkLastDim.h | ||
| unfusedAttentionKernels.cu | ||
| unfusedAttentionKernels.h | ||
| xqaDispatcher.cpp | ||
| xqaDispatcher.h | ||