mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-23 04:03:22 +08:00
0d4d50a745
* init trtllm attn no cache Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * fix: fix the seq_len issue and attn metadata prepare for qwen reward model test fix: fix minor bugs after rebase Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: remove unnecessary debug logs and clean up commented code refactor: update max_seq_len documentation and remove max_seq_len for decoder model contructor in PyTorchModelEngine Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: update calculate_ref_result function to accept tensor inputs and mask type, enhance test_attention_no_cache to support FULL and CAUSAL masks Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: remove unused BERT attention metadata conversion method and add type assertion for no cache attention in PyTorchModelEngine Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: remove use_kv_cache parameter from attention function and related classes, update documentation for KV cache handling Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: implement setAttentionMaskType method for better mask type handling and remove unused conversion function Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: streamline KV cache handling by replacing direct member access with useKVCache method and simplify token per block assignment remove Debug code. Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: Resolve comments for Python code Simplify no cache attention metadata preparation and streamline related attributes in TrtllmAttentionMetadata Removed the private method for converting to no cache attention metadata and integrated its logic into the prepare method. Updated the test for BERT sequence classification to reflect these changes and ensure proper handling of attention metadata. Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * docs: Add is_dummy_attention field to attention metadata for simulation operations Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * refactor: add KVCacheParams to attention backend interface and import relevant metadata classes Updated the attention backend interface to include KVCacheParams and imported TrtllmAttentionMetadata and VanillaAttentionMetadata in model_engine.py for enhanced functionality. Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * fix: fix rebase format issue Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * fix: extend attention mask type handling in MHARunnerFixedParams Added support for additional attention mask types (BIDIRECTIONAL, BIDIRECTIONALGLM, BLOCKSPARSE) in the MHARunnerFixedParams structure to fix the mapping issue between ContextAttentionMaskType and AttentionMaskType Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> * fix: enhance attention mask type handling in TllmGenFmhaRunnerParams Updated the setAttentionMaskType method to include a switch-case structure for better handling of attention mask types, ensuring proper mapping and error handling for invalid types. Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> --------- Signed-off-by: Qixiang Lin <qixiangl@nvidia.com> |
||
|---|---|---|
| .. | ||
| beamSearchKernels | ||
| contextFusedMultiHeadAttention | ||
| cutlass_kernels | ||
| decoderMaskedMultiheadAttention | ||
| flashMLA | ||
| fusedLayernormKernels | ||
| internal_cutlass_kernels | ||
| lora | ||
| selectiveScan | ||
| speculativeDecoding | ||
| trtllmGenKernels | ||
| unfusedAttentionKernels | ||
| userbuffers | ||
| weightOnlyBatchedGemv | ||
| allReduceFusionKernels.cu | ||
| allReduceFusionKernels.h | ||
| attentionMask.cu | ||
| attentionMask.h | ||
| banBadWords.cu | ||
| banBadWords.h | ||
| banRepeatNgram.cu | ||
| banRepeatNgram.h | ||
| beamSearchKernels.cu | ||
| beamSearchKernels.h | ||
| buildRelativeAttentionBiasKernel.cu | ||
| buildRelativeAttentionBiasKernel.h | ||
| CMakeLists.txt | ||
| cumsumLastDim.cu | ||
| cumsumLastDim.h | ||
| customAllReduceKernels.cu | ||
| customAllReduceKernels.h | ||
| decoderMaskedMultiheadAttention.cu | ||
| decoderMaskedMultiheadAttention.h | ||
| decoderMaskedMultiheadAttentionUtils.h | ||
| decodingCommon.cu | ||
| decodingKernels.cu | ||
| decodingKernels.h | ||
| delayStream.cu | ||
| delayStream.h | ||
| doraScaling.cu | ||
| doraScaling.h | ||
| fmhaDispatcher.cpp | ||
| fmhaDispatcher.h | ||
| gptKernels.cu | ||
| gptKernels.h | ||
| groupGemm.cu | ||
| groupGemm.h | ||
| kvCachePartialCopy.cu | ||
| kvCacheUtils.h | ||
| layernormKernels.cu | ||
| layernormKernels.h | ||
| logitsBitmask.cu | ||
| logitsBitmask.h | ||
| lookupKernels.cu | ||
| lookupKernels.h | ||
| lruKernel.cu | ||
| lruKernel.h | ||
| mambaConv1dKernels.cu | ||
| mambaConv1dKernels.h | ||
| mlaKernels.cu | ||
| mlaKernels.h | ||
| multiHeadAttentionCommon.h | ||
| noAuxTcKernels.cu | ||
| noAuxTcKernels.h | ||
| penaltyKernels.cu | ||
| penaltyKernels.h | ||
| penaltyTypes.h | ||
| preQuantScaleKernel.cu | ||
| preQuantScaleKernel.h | ||
| qserveGemm.h | ||
| qserveGemmPerChannel.cu | ||
| qserveGemmPerGroup.cu | ||
| quantization.cu | ||
| quantization.cuh | ||
| quantization.h | ||
| rmsnormKernels.cu | ||
| rmsnormKernels.h | ||
| sageAttentionKernels.cu | ||
| sageAttentionKernels.h | ||
| samplingAirTopPKernels.cu | ||
| samplingTopKKernels.cu | ||
| samplingTopKKernels.h | ||
| samplingTopPKernels.cu | ||
| samplingTopPKernels.h | ||
| splitkGroupGemm.cu | ||
| splitkGroupGemm.h | ||
| stopCriteriaKernels.cu | ||
| stopCriteriaKernels.h | ||
| topkLastDim.cu | ||
| topkLastDim.h | ||
| unfusedAttentionKernels.cu | ||
| unfusedAttentionKernels.h | ||
| xqaDispatcher.cpp | ||
| xqaDispatcher.h | ||