mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-25 05:02:59 +08:00
286a789549
* feat: Add heuristic for GroupRMSNorm kernel selection. Implements a logistic regression model to dynamically select between: - GroupRMSNormBaseKernel: Allocates warps proportional to sum of dimensions (better SM occupancy in most cases) - GroupRMSNormLargeBatch: Allocates warps proportional to max dimension (better block scheduling in large batch scenarios) Selection heuristic considers batch size, allocated warps, and scheduling efficiency on the current GPU architecture. Models for Compute Capability 9.x and 10.x are trained base on nsys kernel runtime data. The default kernel selection is the base kernel. The python operator group_rms_norm will use the heuristic by default. User can pick to use the base or large batch kernels as well. Signed-off-by: Simeng Liu <simengl@nvidia.com> * Address the comments. Signed-off-by: Simeng Liu <simengl@nvidia.com> --------- Signed-off-by: Simeng Liu <simengl@nvidia.com> |
||
|---|---|---|
| .. | ||
| attention_backend | ||
| auto_deploy | ||
| compilation | ||
| custom_ops | ||
| distributed | ||
| models | ||
| modules | ||
| peft | ||
| pyexecutor | ||
| speculative | ||
| __init__.py | ||
| autotuner.py | ||
| llm.py | ||
| metadata.py | ||
| model_config.py | ||
| pipeline_interface.py | ||
| utils.py | ||