Installing on Windows

The Windows release of TensorRT-LLM is currently in beta. We recommend using the `rel` branch for the most stable experience.

Prerequisites

Clone this repository using Git for Windows.
Install the dependencies one of two ways:
1. Install all dependencies together.
  1. Run the provided PowerShell script setup_env.ps1 located under /windows/ folder which installs Python, CUDA 12.4 and cuDNN automatically with default settings. Run PowerShell as Administrator to use the script. Note that cuDNN is installed in the current working directory in which the script is launched.
```
./setup_env.ps1 [-skipCUDA] [-skipPython] [-skipCUDNN]
```
  1. Add cuDNN to your system's Path environment variable by executing
```
[Environment]::SetEnvironmentVariable('Path', $Env:Path + ';' + $Env:CUDNN, [EnvironmentVariableTarget]::Machine)
```
  and closing followed by re-opening any existing PowerShell or Git Bash windows so they pick up the new Path.
2. Install the dependencies one at a time.
  1. Install Python 3.10.
    1. Select Add python.exe to PATH at the start of the installation. The installation may only add the python command, but not the python3 command.
    2. Navigate to the installation path %USERPROFILE%\AppData\Local\Programs\Python\Python310 (AppData is a hidden folder) and copy python.exe to python3.exe.
  2. Install CUDA 12.4 Toolkit. Use the Express Installation option. Installation may require a restart.
  3. [Optional] Download and install Microsoft MPI. You will be prompted to choose between an exe, which installs the MPI executable, and an msi, which installs the MPI SDK. Download and install both.
  4. Download and unzip cuDNN.
    1. Move the folder to a location you can reference later, such as %USERPROFILE%\inference\cuDNN.
    2. Add the libraries and binaries for cuDNN to your system's Path environment variable.
      1. Click the Windows button and search for environment variables.
      2. Click Edit the system environment variables > Environment Variables.
      3. In the new window under System variables, click Path > Edit. Add New lines for the bin and lib directories of cuDNN. Your Path should include lines like this:
```
%USERPROFILE%\inference\cuDNN\bin
%SERPROFILE%\inference\cuDNN\lib
```
      1. Click OK on all the open dialog windows.
      2. Close and re-open any existing PowerShell or Git Bash windows so they pick up the new Path.

Steps

Install TensorRT-LLM.

pip install tensorrt_llm --extra-index-url https://pypi.nvidia.com --extra-index-url https://download.pytorch.org/whl/cu121

Run the following command to verify that your TensorRT-LLM installation is working properly.

python -c "import tensorrt_llm; print(tensorrt_llm._utils.trt_version())"

Build the model.
Deploy the model.

3.4 KiB Raw Blame History

Installing on Windows

3.4 KiB

Raw Blame History