Neta Zmora 34dc6869f3 [#8732][feat] Update TRTLLM Cutlass MoE kernels with ReLU2 (#9011) ... Update TRTLLM Cutlass MoE kernels with ReLU2 activation. Nemotron-6 requires ReLU2 (i.e. squared ReLU) MoE activation function. The PR adds this and adds an API to set the activation function, in general. The ReLU2 changes are based on this FlashInfer PR: https://github.com/flashinfer-ai/flashinfer/pull/1954. The PR also updates the Auto Deploy MoE backend for 16-bit and FP8 from Triton (`torch.ops.auto_deploy.triton_moe_fused`, `torch.ops.auto_deploy.triton_quant_fp8_moe`) to TRTLLM/Cutlass (`torch.ops.auto_deploy.trtllm_moe_fused`, `torch.ops.auto_deploy.trtllm_quant_fp8_moe_fused`). Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>		2025-11-13 16:54:45 -08:00
..
_utils_test	[None][chore] AutoDeploy: replace HF's deprecated keyword torch_dtype --> dtype (#8510)	2025-10-21 17:07:06 -04:00
unit	[#8732][feat] Update TRTLLM Cutlass MoE kernels with ReLU2 (#9011)	2025-11-13 16:54:45 -08:00