mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00

History

Yuan Tong 4b6c19737b feat: support add internal cutlass kernels as subproject (#3658 ) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com>		2025-05-06 11:35:07 +08:00
..
CMakeLists.txt	Update TensorRT-LLM (#2008 )	2024-07-23 23:05:09 +08:00
gen-moe-benchmark-file.py	Update TensorRT-LLM (#2873 )	2025-03-11 21:13:42 +08:00
mixtureOfExpertsBackendBenchmarkFixture.h	feat: support add internal cutlass kernels as subproject (#3658 )	2025-05-06 11:35:07 +08:00
mixtureOfExpertsBackendBenchmarkLauncher.cu	Update TensorRT-LLM (#2873 )	2025-03-11 21:13:42 +08:00
README.md	Update TensorRT-LLM (#1891 )	2024-07-04 14:37:19 +08:00

README.md

Micro Benchmarks

This folder contains benchmarks for specific components in TRT-LLM, using google-benchmark

Building

To build add the --micro_benchmark flag to build_wheel.py or pass -DBUILD_MICRO_BENCHMARKS=ON to cmake

Benchmark Documentations

Mixture Of Experts Backend Benchmark

Target mixtureOfExpertsBackendBenchmark

This benchmark covers the backend used by the MixtureOfExperts plugin. It allows you to benchmark different MOE configurations without building a TRT engine.

Usage:

./mixtureOfExpertsBackendBenchmark

# or

./mixtureOfExpertsBackendBenchmark --input_file <JSON benchmark definition>

For more information see:

./mixtureOfExpertsBackendBenchmark --help

The gen-moe-workload-file.py is a helper script that can generate workload files for MOE benchmarks. This is useful for sharing or comparing configurations, such as when generating a reproduction case for a performance bug