mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-02-07 03:31:58 +08:00
148 lines
6.4 KiB
Markdown
148 lines
6.4 KiB
Markdown
# The Docker Build System
|
|
|
|
## Multi-stage Builds with Docker
|
|
|
|
TensorRT-LLM can be compiled in Docker using a multi-stage build implemented in [`Dockerfile.multi`](Dockerfile.multi).
|
|
The following build stages are defined:
|
|
|
|
* `devel`: this image provides all dependencies for building TensorRT-LLM.
|
|
* `wheel`: this image contains the source code and the compiled binary distribution.
|
|
* `release`: this image has the binaries installed and contains TensorRT-LLM examples in `/app/tensorrt_llm`.
|
|
|
|
## Building Docker Images with GNU `make`
|
|
|
|
The GNU [`Makefile`](Makefile) in the `docker` directory provides targets for building, pushing, and running each stage
|
|
of the Docker build. The corresponding target names are composed of two components, namely, `<stage>` and `<action>`
|
|
separated by `_`. The following actions are available:
|
|
|
|
* `<stage>_build`: builds the docker image for the stage.
|
|
* `<stage>_push`: pushes the docker image for the stage to a docker registry (implies `<stage>_build`).
|
|
* `<stage>_run`: runs the docker image for the stage in a new container.
|
|
|
|
For example, the `release` stage is built and pushed from the top-level directory of TensorRT-LLM as follows:
|
|
|
|
```bash
|
|
make -C docker release_push
|
|
```
|
|
|
|
Note that pushing the image to a docker registry is optional. After building an image, run it in a new container with
|
|
```bash
|
|
make -C docker release_run
|
|
```
|
|
|
|
### Building and Running Options
|
|
|
|
The full image name and tag can be controlled by supplying `IMAGE_WITH_TAG` to `make`:
|
|
|
|
```bash
|
|
make -C docker devel_push IMAGE_WITH_TAG="urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:dev"
|
|
```
|
|
|
|
Containers can be started with the local user instead of `root` by appending `LOCAL_USER=1` to the run target:
|
|
|
|
```bash
|
|
make -C docker devel_run LOCAL_USER=1
|
|
```
|
|
|
|
Specific CUDA architectures supported by the `wheel` can be specified with `CUDA_ARCHS`:
|
|
|
|
```bash
|
|
make -C docker release_build CUDA_ARCHS="80-real;90-real"
|
|
```
|
|
|
|
The `run` action maps the locally checked out source code into the `/code/tensorrt_llm` directory within the container.
|
|
|
|
The `DOCKER_RUN_ARGS` option can be used to pass additional options to Docker,
|
|
e.g., in order to mount additional volumes into the container.
|
|
|
|
For more build options, see the variables defined in [`Makefile`](Makefile).
|
|
|
|
### NGC Integration
|
|
|
|
When building from source, one can conveniently download a docker image for development from
|
|
the [NVIDIA NGC Catalog](https://catalog.ngc.nvidia.com/) and start it like so:
|
|
|
|
```bash
|
|
make -C docker ngc-devel_run LOCAL_USER=1 DOCKER_PULL=1
|
|
```
|
|
|
|
As before, specifying `LOCAL_USER=1` will run the container with the local user's identity. Specifying `DOCKER_PULL=1`
|
|
is optional, but it will pull the latest image from the NGC Catalog.
|
|
|
|
We also provide an image with pre-installed binaries for release. This can be used like so:
|
|
|
|
```bash
|
|
make -C docker ngc-release_run LOCAL_USER=1 DOCKER_PULL=1
|
|
```
|
|
|
|
If you want to deploy a specific version of TensorRT-LLM, you can specify the version with
|
|
`IMAGE_TAG=<version_tag>` (cf. [release history on GitHub](https://github.com/NVIDIA/TensorRT-LLM/releases) and [tags in NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags)). The application examples and benchmarks are installed
|
|
in `/app/tensorrt_llm`.
|
|
|
|
See the description of the `<stage>_run` make target in
|
|
[Building and Running Options](#building-and-running-options) for additional information and
|
|
running options.
|
|
|
|
If you cannot access the NGC container images, you can instead locally build and use
|
|
equivalent containers as [described above](#building-docker-images-with-gnu-make).
|
|
|
|
### Jenkins Integration
|
|
|
|
[`Makefile`](Makefile) has special targets for building, pushing and running the Docker build image used on Jenkins.
|
|
The full image name and tag is defined in [`L0_MergeRequest.groovy`](../jenkins/L0_MergeRequest.groovy). The `make`
|
|
system will parse this name as the value of `LLM_DOCKER_IMAGE`. To build and push a new Docker image for Jenkins,
|
|
define a new image name and tag in [`L0_MergeRequest.groovy`](../jenkins/L0_MergeRequest.groovy) and run
|
|
|
|
```bash
|
|
make -C docker jenkins_push
|
|
```
|
|
|
|
Start a new container using the same image as Jenkins using your local user account with
|
|
|
|
```bash
|
|
make -C docker jenkins_run LOCAL_USER=1
|
|
```
|
|
|
|
If you do not have access to the [internal artifact repository](https://urm.nvidia.com/artifactory/sw-tensorrt-docker/tensorrt-llm/), you can instead either use the [NGC Develop
|
|
image](#ngc-integration) or [build an image locally](#building-docker-images-with-gnu-make).
|
|
|
|
#### Release images based on Jenkins image
|
|
|
|
One may also build a release image based on the Jenkins development image:
|
|
|
|
```bash
|
|
make -C docker trtllm_build CUDA_ARCHS="80-real;90-real"
|
|
```
|
|
|
|
Note that the above requires access to the Jenkins development image from the
|
|
[internal artifact repository](https://urm.nvidia.com/artifactory/sw-tensorrt-docker/tensorrt-llm/).
|
|
|
|
The resulting images can be pushed to
|
|
the [internal artifact repository](https://urm.nvidia.com/artifactory/sw-tensorrt-docker/tensorrt-llm-staging/release/):
|
|
|
|
```bash
|
|
make -C docker trtllm_push
|
|
```
|
|
|
|
Generally, only images built for all CUDA architectures should be pushed to the artifact repository. These images can
|
|
be deployed in docker in the usual way:
|
|
|
|
```bash
|
|
make -C docker trtllm_run LOCAL_USER=1 DOCKER_PULL=1
|
|
```
|
|
|
|
The argument `DOCKER_PULL=1` instructs `make` to pull the latest version of the image before deploying it in the container.
|
|
By default, the release images built in the above manner are tagged by their `git` branch name and may be frequently updated.
|
|
|
|
### Docker rootless
|
|
|
|
Some aspects require special treatment when using [Docker rootless mode](https://docs.docker.com/engine/security/rootless/). The `docker/Makefile` contains heuristics to detect Docker rootless mode. When assuming
|
|
Docker rootless mode, the `%_run` targets in `docker/Makefile` will output
|
|
a corresponding message. The heuristics can be overridden by specifying
|
|
`IS_ROOTLESS=0` or `IS_ROOTLESS=1`, respectively.
|
|
|
|
Since Docker rootless mode remaps the UID/GID and the remapped UIDs and GIDs
|
|
(typically configured in `/etc/subuid` and `/etc/subgid`) generally do not coincide
|
|
with the local UID/GID, both IDs need to be translated using a tool like `bindfs` in order to be able to smoothly share a local working directory with any containers
|
|
started with `LOCAL_USER=1`. In this case, the `SOURCE_DIR` and `HOME_DIR` Makefile variables need to be set to the locations of the translated versions of the TensorRT-LLM working copy and the user home directory, respectively.
|