TensorRT-LLMs/docs/source/dev-on-cloud/dev-on-runpod.md

(dev-on-runpod)=

Develop TensorRT-LLM on Runpod

Runpod is a popular cloud platform among many researchers. This doc describes how to develop TensorRT-LLM on Runpod.

Prepare

Create a Runpod account

Please refer to the Runpod Getting Started.

Configure SSH Key

Please refer to the Configure SSH Key.

Note that we can skip the step of "Start your Pod. Make sure of the following things" here as we will introduce it below.

Build the TensorRT-LLM Docker Image and Upload to DockerHub

Please refer to the Build Image to DockerHub.

Note that the docker image must enable ssh access. See on Enable ssh access to the container.

Create a Pod Template

Click "Template" bottom on the menus and click "Create Template" bottom.

Fill the docker image link of DockerHub such as docker.io/<your_dockerhub_username>/tensorrt_llm:devel on "Docker Image" field.

Fill "22" into "Expose TCP Ports" field.

Fill

sleep infinity

into 'Container Start Command' field.

Connect to the Pod

Please refer to the Connect to the Pod.

You can connect the pod with SSH or Web Terminal.

If you want to connect the pod with SSH, you can copy the command from "SSH over exposed TCP" field and run it on your host.

In some scenarios such as using a team account, your public key has not been added to the pod successfully. You can directly add this command to the 'Container Start Command' field as:

bash -c 'echo "<your_public_key>" >> ~/.ssh/authorized_keys;sleep infinity'

Enjoy your development!