kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-02-06 19:21:52 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
0d5ffae9a7
TensorRT-LLMs/benchmarks/README.md
Kaiyu Xie bca9a33b02
Update TensorRT-LLM (#2008) 
* Update TensorRT-LLM

---------

Co-authored-by: Timur Abishev <abishev.timur@gmail.com>
Co-authored-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>
Co-authored-by: Saeyoon Oh <saeyoon.oh@furiosa.ai>
Co-authored-by: hattizai <hattizai@gmail.com>
2024-07-23 23:05:09 +08:00

561 B
Raw Blame History

TensorRT-LLM Benchmarks

Overview

There are currently three workflows to benchmark TensorRT-LLM:

  • C++ benchmarks
    • The recommended workflow that uses TensorRT-LLM C++ API and can take advantage of the latest features of TensorRT-LLM.
  • Python benchmarks
    • The Python benchmarking scripts can only benchmark the Python runtime, which do not support the latest features, such as in-flight batching.
  • The Python benchmarking suite
    • This benchmarking suite is a current work in progress and is prone to large changes.