Welcome to TensorRT-LLM’s documentation!

Contents:

• TensorRT-LLM Architecture
• C++ GPT Runtime
• The Batch Manager in TensorRT-LLM
• Multi-head, Multi-query and Group-query Attention
• Numerical Precision
• Performance of TensorRT-LLM
• Build From Sources
• How to debug
• How to add a new model
• Graph Rewriting Module

Python API

• tensorrt_llm.layers

• tensorrt_llm.functional

• tensorrt_llm.models

• tensorrt_llm.plugin

• tensorrt_llm.Quantization

• tensorrt_llm.runtime

C++ API

• cpp/runtime

Indices and tables

• Index

• Module Index

• Search Page