mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-13 22:18:36 +08:00

History

Kaiyu Xie aaacc9bd68 Update TensorRT-LLM (#2562 ) * Update TensorRT-LLM --------- Co-authored-by: Starrick Liu <73152103+StarrickLiu@users.noreply.github.com>		2024-12-11 00:31:05 -08:00
..
cpp	Update TensorRT-LLM (#2562 )	2024-12-11 00:31:05 -08:00
python	Update TensorRT-LLM (#2532 )	2024-12-04 21:16:56 +08:00
README.md	Update TensorRT-LLM (#2413 )	2024-11-05 16:27:06 +08:00

TensorRT-LLM Benchmarks

Overview

There are currently three workflows to benchmark TensorRT-LLM:

C++ benchmarks
- The recommended workflow that uses TensorRT-LLM C++ API and can take advantage of the latest features of TensorRT-LLM.
Python benchmarks
- The Python benchmarking scripts can only benchmark the Python runtime, which do not support the latest features, such as in-flight batching.
The Python benchmarking suite
- This benchmarker is native to TensorRT-LLM and is a Python benchmarker for reproducing and testing the performance of TensorRT-LLM.
- NOTE: This benchmarking suite is a current work in progress and is prone to large changes.