mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-02-12 14:03:48 +08:00
34dc6869f3
Update TRTLLM Cutlass MoE kernels with ReLU2 activation. Nemotron-6 requires ReLU2 (i.e. squared ReLU) MoE activation function. The PR adds this and adds an API to set the activation function, in general. The ReLU2 changes are based on this FlashInfer PR: https://github.com/flashinfer-ai/flashinfer/pull/1954. The PR also updates the Auto Deploy MoE backend for 16-bit and FP8 from Triton (`torch.ops.auto_deploy.triton_moe_fused`, `torch.ops.auto_deploy.triton_quant_fp8_moe`) to TRTLLM/Cutlass (`torch.ops.auto_deploy.trtllm_moe_fused`, `torch.ops.auto_deploy.trtllm_quant_fp8_moe_fused`). Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| attention | ||
| auto_deploy | ||
| compilation | ||
| debugger | ||
| executor | ||
| misc | ||
| modeling | ||
| models/checkpoints/hf | ||
| modules | ||
| multi_gpu | ||
| multi_gpu_modeling | ||
| multimodal | ||
| ray_orchestrator | ||
| sampler | ||
| speculative | ||
| thop | ||
| helpers.py | ||
| pattern_watcher.py | ||
| test_connector.py | ||