Performance
Public Functions
+Constructs a DisaggExecutorOrchestrator object.
+ctxEnginePaths – A vector of file paths to context engine files.
genEnginePaths – A vector of file paths to generation engine files.
ctxExecutorConfigs – A vector of ExecutorConfig for context executors.
genExecutorConfigs – A vector of ExecutorConfig for generation executors.
hasContextAwaitThreads – Whether or not there are threads that receive response for each generation executor.
hasGenAwaitThreads – Whether or not there are threads that receive response for each generation executor.
Enqueue context-only requests to context executors.
+requests – A vector of context-only requests.
selectContextId – The index of the context executor to use. If std::nullopt, the executor that has the smallest number of inflight requests will be used.
batch – If true,enqueue requests in same context executor.If false, will try to use a different executor for each request.
A vector of global request ids, corresponding to the order of the requests in requests, the id returned may be different from the request id in each executor.
Enqueue generation-only requests to generation executors.
+requests – A vector of generation-only requests.
globalRequestIds – A vector of global request ids, corresponding to the order of the requests,and must be the ids returned by the enqueueContext function.
selectGenIdx – The index of the generation executor to use. If std::nullopt, the executor that has the smallest number of inflight requests will be used.
batch – If true,enqueue requests in same generation executor.If false, will try to use a different executor for each request.
Await for context responses.
+timeout – The maximum time to wait for new responses
contextIdx – The index of the context executor to use. If std::nullopt, return ready responses in all context executors,if hasContextAwaitThreads is true, then this parameter must be std::nullopt.
A vector of responses with corresponding global request ids
+Await for generation responses.
+timeout – The maximum time to wait for new responses.
genIdx – The index of the generation executor to use. If std::nullopt, return ready responses in all generation executors,if hasGenAwaitThreads is true, then this parameter must be std::nullopt.
A vector of responses with corresponding global request ids.
+Indicates if the current process is allowed to enqueueRequests.
+Get context executors.
+Get generation executors.
+Private Members
+Public Functions
+Typedefs
+Enums
+ + + + +Functions
+Public Functions
+Private Members
+Public Functions
+Public Functions
+Public Static Functions
+Private Members
+Private Static Functions
+Public Functions
+Public Static Functions
+Public Functions
+Public Functions
+Private Members
+Public Static Functions
+Public Functions
-Constructs a DisaggExecutorOrchestrator object.
-ctxEnginePaths – A vector of file paths to context engine files.
genEnginePaths – A vector of file paths to generation engine files.
ctxExecutorConfigs – A vector of ExecutorConfig for context executors.
genExecutorConfigs – A vector of ExecutorConfig for generation executors.
hasContextAwaitThreads – Whether or not there are threads that receive response for each generation executor.
hasGenAwaitThreads – Whether or not there are threads that receive response for each generation executor.
Enqueue context-only requests to context executors.
-requests – A vector of context-only requests.
selectContextId – The index of the context executor to use. If std::nullopt, the executor that has the smallest number of inflight requests will be used.
batch – If true,enqueue requests in same context executor.If false, will try to use a different executor for each request.
A vector of global request ids, corresponding to the order of the requests in requests, the id returned may be different from the request id in each executor.
Enqueue generation-only requests to generation executors.
-requests – A vector of generation-only requests.
globalRequestIds – A vector of global request ids, corresponding to the order of the requests,and must be the ids returned by the enqueueContext function.
selectGenIdx – The index of the generation executor to use. If std::nullopt, the executor that has the smallest number of inflight requests will be used.
batch – If true,enqueue requests in same generation executor.If false, will try to use a different executor for each request.
Await for context responses.
-timeout – The maximum time to wait for new responses
contextIdx – The index of the context executor to use. If std::nullopt, return ready responses in all context executors,if hasContextAwaitThreads is true, then this parameter must be std::nullopt.
A vector of responses with corresponding global request ids
-Await for generation responses.
-timeout – The maximum time to wait for new responses.
genIdx – The index of the generation executor to use. If std::nullopt, return ready responses in all generation executors,if hasGenAwaitThreads is true, then this parameter must be std::nullopt.
A vector of responses with corresponding global request ids.
-Indicates if the current process is allowed to enqueueRequests.
-Get context executors.
-Get generation executors.
-Private Members
-Public Functions
-Public Functions
-Private Members
-Friends
-Public Functions
-Public Types
- - -Public Functions
-Private Members
-Friends
-Public Functions
-Public Functions
-Public Members
-Public Functions
-Public Members
-Public Functions
-Private Members
-Friends
-Public Functions
- - -Public Members
-Public Functions
-Public Static Functions
+Public Functions
Private Members
+Public Functions
+Public Types
+ + +Public Functions
+Private Members
+Friends
+Public Functions
+Public Functions
+Public Members
+Public Functions
+Public Members
+Public Functions
+Private Members
+Friends
+Public Functions
+ + +Public Members
+Public Functions
+Typedefs
-Enums
- - - - -Functions
-Public Functions
-Private Members
-Public Functions
-Public Functions
-Public Static Functions
-Private Members
-Private Static Functions
-Public Functions
-Public Static Functions
-Public Functions
-Public Functions
-Private Members
-Performance
Public Functions
Public Static Functions
-Private Members
-Public Functions
-Is my rank the last rank in its pipeline?
-Public Static Functions
-Public Static Attributes
-Private Members
-Defines
-Typedefs
-Enums
- - -Public Functions
-Public Members
Functions
-Public Functions
-Public Types
+ +Public Functions
Public Static Attributes
-Private Functions
-GPT decoder class with support for in-flight batching.
-Subclassed by tensorrt_llm::runtime::GptDecoderBatched
-Public Types
-Public Functions
-Setup the decoder before calling forward()
Disable Lookahead decoding.
-Run one step for all requests without blocking the host process and return the token for synchronization.
-Run one step for all requests and wait for completion on the host.
-Gather final beam search results for request batchIdx. Result will only be available after event returned.
Public Types
-Public Functions
-Public Members
-[maxDecoderSteps][batchSize][1, beamWidth, vocabSizePadded], on gpu
-Mandatory parameters Logits
-Maximum number of decoding tokens of active slots.
-Public Types
-Public Functions
-Public Members
- - -Private Functions
-Private Members
-Public Members
- - - - - - - - - - - - - - -Public Functions
-Public Members
- - - - - - - - -[maxBatchSize, maxDecodingTokens] or [numSequences, maxDecodingTokens]
-[maxBatchSize, maxDecodingDraftTokens] or [numSequences, maxDecodingDraftTokens]
-[maxBatchSize, maxNumPaths, maxPathLen] or [numSequences, maxNumPaths, maxPathLen]
-[maxBatchSize, maxNumPaths, maxPathLen] or [numSequences, maxNumPaths, maxPathLen]
-[maxBatchSize, maxDecodingTokens, ceil(maxDecodingTokens / 32)] or [numGenSequences, maxDecodingTokens, ceil(maxDecodingTokens / 32)]
-[maxBatchSize * maxDecodingTokens] or [numSequences * maxDecodingTokens]
-[maxBatchSize, maxDecodingDraftTokens] or [numSequences, maxDecodingDraftTokens]
-[maxBatchSize, maxDecodingDraftTokens] or [numSequences, maxDecodingDraftTokens]
-[maxBatchSize, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens] or [numSequences, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens]
-Defines
-Public Functions
-Public Members
-Private Types
-Private Functions
- - -Public Types
-Public Functions
-Public Static Functions
-Private Functions
-Private Members
-Private Static Attributes
-Public Functions
-Public Static Functions
-Private Members
-Public Functions
-Public Members
-Public Functions
-Public Members
-Mandatory parameters Previously generated token ids for all steps before DecodingInput.step, [BS, BM, MSL]
-The tokens computed during the gatherTree step, [BS, BM, MSL] Necessary for “Streaming + Beam Search” mode since beam search kernels store ungathered tokens in ids.
New tokens at each generated token of maxTokensPerStep, [maxTokensPerStep, BS, BM].
-A Vector of views on newTokensSteps for each token [BS, BM].
-Optional parameters FinishedState by decoding if any of the stop conditions are met or if DecodingInput.finished is true, [BS, BM]
-Mandatory parameters for Beam Search log-probility of generated tokens, [BS, BM, MSL], float
-Public Static Attributes
-Public Functions
-Public Functions
-Public Members
-Public Types
- - -Public Functions
-Setup buffers for the decoder excluding speculative decoding.
-Setup buffers for the cache indirection.
-This is used for beam search on pipeline parallel ranks without a decoder.
-Setup buffers for speculative decoding.
-Disable lookahead decoding.
-[batchSize], number of finished sequences per request, on gpu
-[batchSize, beamWidth], FinishedState value, on gpu
-[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding, on gpu. In case of beam search, contains the ungathered data.
-batchIdx – index of the batch
-[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request batchIdx, on gpu. In case of beam search, contains the ungathered data.
[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], only used for beam search. It contains gathered token ids without padding, on gpu.
-batchIdx – index of the batch
-[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], only used for beam search. It contains gathered token ids without padding for request batchIdx, on gpu.
[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains parent ids collected during beam search without padding, on gpu
-[batchSize, maxBeamWidth], cumulative log probabilities (per beam), on gpu
-[maxBeamWidth], cumulative log probabilities (per beam), on gpu
-[batchSize, maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu
-[maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu
-[batchSize, maxBeamWidth], sequence lengths, on gpu
-batchIdx – index of the batch
-[maxBeamWidth], sequence lengths for request batchIdx, on gpu
Get maxTokensPerStep tokens generated in the last forward pass.
-[maxTokensPerStep, batchSize, maxBeamWidth], tokens generated in last forward pass, on gpu
-[batchSize, maxDraftTokens], predicted draft tokens for next step, on gpu
-[batchSize], predicted draft tokens lengths for previous step, on gpu
-[batchSize], predicted draft tokens lengths for next step, on gpu
-[batchSize + 1], exclusive sum of accepted draft token lengths, on gpu
-[batchSize, maxAcceptedDraftTokensPerStep], accepted paths packed into continuous tensor, on gpu
-[maxTokensPerStep, batchSize, beamWidth], finished states of type FinishedState, on gpu
-Get the number of tokens for all requests in the batch.
-The number of tokens for all requests in the batch.
-Get the number of tokens for a specific request in the batch.
-batchIdx – The index of the request in the batch.
-The number of tokens for the specified request.
-Set the number of tokens for a specific request in the batch.
-batchIdx – The index of the request in the batch.
numTokens – The number of tokens for the specified request.
Get the speculative decoding mode.
-Get the explicit draft tokens buffers.
-Get the eagle buffers.
-Get the lookahead buffers.
-Workspace for beam search in streaming mode.
-Get the generation steps for all requests in the batch.
-The generation steps for all requests in the batch.
-Set the generation steps for all requests in the batch.
-generationSteps – The generation steps for all requests in the batch.
-Stateful inputs for the decoder. Allocated for maxBatchSize slots.
-Stateful outputs for the decoder. Allocated for maxBatchSize slots.
-Private Functions
-Private Members
-Stateful inputs for the decoder. Allocated for maxBatchSize slots.
-Stateful outputs for the decoder. Allocated for maxBatchSize slots.
-[maxTokensPerStep, batchSize, beamWidth] finished states of type FinishedState for each generated token of maxTokensPerStep, on gpu
-Workspace for beam search in streaming mode.
-[batchSize], the num tokens of each request.
-Functions
-Helper function to produce batch slots [0, 1, …, batchSize - 1] for paths that do not explicitly provide batch slots to the decoder.
-Public Types
-Public Functions
-explicitDraftTokensDType – is only used by ExplicitDraftTokens model to WAR the lack of bf16 decoder.
-Private Members
-Subclassed by tensorrt_llm::runtime::GptDecoder< T >
-Public Types
- - -Public Functions
-explicitDraftTokensDType – is only used by ExplicitDraftTokens model to WAR the lack of bf16 decoder.
-Public Static Functions
-Public Types
-Public Functions
-Public Members
Private Functions
-Public Members
- - - - - - - - - - - - - - - - - - - - - - - - -Subclassed by tensorrt_llm::runtime::ExplicitDraftTokensBuffers::EngineInputs
-Public Functions
-Public Members
- - - - - - - - -[maxBatchSize, maxNumPaths, maxPathDraftLen] or [numGenSequences, maxNumPaths, maxPathDraftLen]
-[maxBatchSize, maxNumPaths, maxPathLen] or [numGenSequences, maxNumPaths, maxPathLen]
-[maxBatchSize, maxNumPaths, maxPathLen] or [numGenSequences, maxNumPaths, maxPathLen]
-[maxBatchSize, maxNumPaths, maxPathDraftLen, vocabSize] or [numGenSequences, maxNumPaths, maxPathDraftLen, vocabSize]
-Represents the inputs to the decoder.
-This input type is assumed immutable. It represents whatever the decoder received initially, and can always be referred to as such.
-Public Types
-Public Functions
-Public Members
-Mandatory parameters The index of the decoding step we are on. Only used in Python runtime
-The maximum number of tokens to decode.
-The maximum length of the attention window to consider while decoding.
-The number of tokens to use as attention sinks, https://arxiv.org/html/2309.17453v3.
-The number of samples in the batch.
-The beam widths of each request, [batchSize].
-The maximum value in the stopWordsLens tensor.
The maximum value in the badWordsLens tensor.
The output of the model forward computation, a probability distribution over the vocabulary [batchSize][numGenTokens, beamWidth, vocabSizePadded] on gpu
-The end ids, [batchSize * beamWidth] on gpu.
-Address map of the linear batch id to to the seq slots, [batchSize] on pinned, int32_t.
-Optional parameters Finished states at current iteration (skip decoding step of a request if true), [batchSize, beamWidth] on gpu
-The maximum sequence length for each sequence in the batch, [batchSize] on gpu.
-Parameters for beam search KV cache index for beam search, [batchSize, beamWidth, maxSeqLen] on gpu
-Steps of each request, for Variable-Beam-Width-Search, [batchSize].
-Public Members
-Public Members
-Public Members
- - - - - - - - - - - - - - - - -Public Members
-[batchSize, maxTokensPerStep, maxMedusaHeads + 1], on gpu
-[batchSize, maxTokensPerStep], on gpu
-[batchSize][maxAcceptedDraftTokensPerStep][maxDraftTokens + 1, vocabSizePadded], on gpu
-[batchSize], on gpu
-A helper class for managing memory on host and device.
-Public Types
+Typedefs
- - - -Public Functions
+Enums
+ + +Functions
Construct a BufferManager.
+Gets a typed pointer to the constant underlying data of the buffer.
cudaStream – [in] The cuda stream to use for all operations on GPU (allocation, de-allocation, copying, etc.).
+T – The type of the underlying data.
+buffer – The buffer to get a pointer to.
+A pointer to constant T.
Destructor.
+Gets a typed pointer to the underlying data of the buffer.
+T – The type of the underlying data.
+buffer – The buffer to get a pointer to.
+A pointer to T.
Allocates an IBuffer of the given size on the GPU, using cudaMallocAsync.
Retrieves a T typed pointer to the underlying data of the buffer pointed to by the bufferPtr, or nullptr if the bufferPtr is null.
+T – The type of the underlying data.
+bufferPtr – A possibly null shared ptr.
+A pointer to T, possibly nullptr.
+Allocates an ITensor of the given dimensions on the GPU, using cudaMallocAsync.
Retrieves a T const typed pointer to the underlying data of the buffer pointed to by the bufferPtr, or nullptr if the bufferPtr is null.
+T – The type of the underlying data.
+bufferPtr – A possibly null shared ptr.
+A pointer to const T, possibly nullptr.
+Allocates an IBuffer of the given size and memory type.
Retrieves a T typed pointer to the underlying data of the buffer pointed to by the buffer pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
+T – The type of the underlying data.
+optionalBufferPtr – A possibly empty optional.
+A pointer to T, possibly nullptr.
+Allocates an ITensor of the given dimensions and memory type.
Retrieves a T const typed pointer to the underlying data of the buffer pointed to by the buffer pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
+T – The type of the underlying data.
+optionalBufferPtr – A possibly empty optional.
+A pointer to const T, possibly nullptr.
+Create an empty IBuffer of the given memory type. It may be resized later.
Create an empty ITensor of the given memory type. It may be reshaped later.
Set the contents of the given buffer to value.
Copy src to dst.
Copy src to dst.
Copy src into a new IBuffer with a potentially different memory type.
Copy src into a new ITensor with a potentially different memory type.
Copy src into a new IBuffer with a potentially different memory type.
Copy src into a new ITensor with a potentially different memory type.
Copy src into a new ITensor with a potentially different memory type.
Get the underlying cuda stream.
-The current size of the memory reserved by the memory pool.
-The current size of the memory used by the memory pool.
-The current size of the memory free in the memory pool.
-A wrapper around nvinfer1::DataType that provides a support for pointer types.
Public Static Functions
+Public Functions
Allocates an IBuffer of the given size on the GPU, using cudaMalloc.
Allocates an ITensor of the given dimensions on the GPU, using cudaMalloc.
Allocates an IBuffer of the given size on the CPU.
Allocates an ITensor of the given dimensions on the CPU.
Allocates a pinned IBuffer of the given size on the CPU.
Allocates a pinned ITensor of the given dimensions on the CPU.
Allocates a pinned IBuffer of the given size on the CPU in the default memory pool.
Allocates a pinned ITensor of the given dimensions on the CPU in the default memory pool.
Allocates an IBuffer of the given size in UVM.
Allocates an ITensor of the given dimensions in UVM.
Allocates an ITensor of the given dimensions for NVLS.
Public Static Attributes
Private Members
Friends
-Public Types
- - -Public Functions
-Functions
-Public Types
-Values:
-Public Functions
+ +For converting a TensorRT data type to a C++ data type.
+Public Types
+Public Types
+Public Types
+Public Types
+Public Types
+Public Types
+Public Types
+Public Types
+Public Types
+Subclassed by tensorrt_llm::runtime::ITensor
+Public Types
+ + + + + + + + + + +Public Functions
+Returns a pointer to underlying array.
+Returns a pointer to underlying array.
+Returns a pointer to the underlying array at a given element index.
+Returns a pointer to the underlying array at a given element index.
+Returns the size (in number of elements) of the buffer.
+Returns the size (in bytes) of the buffer.
+Returns the capacity of the buffer.
+Returns the memory type of the buffer.
+Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity.
+Releases the buffer. It will be reset to nullptr.
+Not allowed to copy.
+Not allowed to copy.
+Public Static Functions
Private Members
-Public Types
-Public Functions
-Public Members
-Mandatory parameters.
-Public Functions
-Creates a new cuda stream on the current device. The stream will be destroyed in the destructor.
+)#Creates a sliced view on the underlying buffer. The view will have the same data type as buffer.
flags – Flags for stream creation. See cudaStreamCreateWithFlags for a list of valid flags that can be passed.
priority – Priority of the stream. Lower numbers represent higher priorities. See cudaDeviceGetStreamPriorityRange for more information about the meaningful stream priorities that can be passed.
buffer – The buffer to view.
offset – The offset of the view.
size – The size of the view.
A view on the buffer.
Pass an existing cuda stream to this object.
+)#Returns a view on the underlying tensor which can be independently resized.
tensor – The tensor to view.
+A view on the tensor.
Returns a view on the underlying tensor with a different size.
stream – The stream to pass to this object.
device – The device on which the stream was created.
ownsStream – Whether this object owns the stream and destroys it in the destructor.
tensor – The tensor to view.
size – The size of the view.
Construct with an existing cuda stream or the default stream by passing nullptr.
-Returns the device on which the stream was created.
-Returns the stream associated with this object.
-Synchronizes the stream.
-Private Types
- - -Public Types
-Public Functions
-Creates a new cuda event. The event will be destroyed in the destructor.
-flags – Flags for event creation. By default, event timing is disabled.
+A view on the tensor.
Pass an existing cuda event to this object.
+Wraps the given data in an IBuffer. The IBuffer will not own the underlying data and cannot be resized beyond capacity.
event – The event to pass to this object.
ownsEvent – Whether this object owns the event and destroys it in the destructor.
data – The data to wrap.
type – The data type of the data.
size – The size of the buffer.
capacity – The capacity of the buffer.
An IBuffer.
Synchronizes the event.
+Determine the memory type of a pointer.
Private Types
-Public Static Attributes
+Public Static Attributes
+Public Static Attributes
+Public Static Attributes
+For converting a C++ data type to a TensorRT data type.
+Public Static Attributes
+Public Static Attributes
+Public Static Attributes
+Public Static Attributes
+Public Static Attributes
+Public Static Attributes
+Private Static Attributes
+Public Static Attributes
+Public Functions
+Public Members
+Mandatory parameters Previously generated token ids for all steps before DecodingInput.step, [BS, BM, MSL]
+The tokens computed during the gatherTree step, [BS, BM, MSL] Necessary for “Streaming + Beam Search” mode since beam search kernels store ungathered tokens in ids.
New tokens at each generated token of maxTokensPerStep, [maxTokensPerStep, BS, BM].
+A Vector of views on newTokensSteps for each token [BS, BM].
+Optional parameters FinishedState by decoding if any of the stop conditions are met or if DecodingInput.finished is true, [BS, BM]
+Mandatory parameters for Beam Search log-probility of generated tokens, [BS, BM, MSL], float
+Public Static Attributes
+Public Functions
+Functions
-Public Types
+ -Public Functions
-Private Members
+Public Functions
+ + +Public Members
-Public Types
+ -Public Functions
+Functions
-Utility function to print a shape.
-Utility function to print a tensor with its shape.
-Retrieves a T const typed pointer to the underlying data of the tensor pointed to by the tensorPtr, or nullptr if the tensorPtr is null.
-This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
-T – The type of the underlying data.
-tensorPtr – A possibly null shared ptr.
-A pointer to T const, possibly nullptr.
-Retrieves a T typed pointer to the underlying data of the buffer pointed to by the tensorPtr, or nullptr if the tensorPtr is null.
-This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
-T – The type of the underlying data.
-tensorPtr – A possibly null shared ptr.
-A pointer to T, possibly nullptr.
-Retrieves a T typed pointer to the underlying data of the tensor pointed to by the tensor pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
-This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
-T – The type of the underlying data.
-optionalBufferPtr – A possibly empty optional.
-A pointer to T, possibly nullptr.
-Retrieves a T const typed pointer to the underlying data of the tensor pointed to by the tensor pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
-This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
-T – The type of the underlying data.
-optionalBufferPtr – A possibly empty optional.
-A pointer to const T, possibly nullptr.
-A helper class for managing memory on host and device.
+Public Types
Public Functions
Returns the tensor dimensions.
+Construct a BufferManager.
+cudaStream – [in] The cuda stream to use for all operations on GPU (allocation, de-allocation, copying, etc.).
+Returns the tensor n-th dimension. If n is negative, returns the (nbDims - n)th dimension. TODO: replace with constexpr parameter when moving to C++20.
+Destructor.
Sets the tensor dimensions. The new size of the tensor will be volume(dims)
Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity.
-Removes the given unit dimensions from this tensor.
-Adds a unit dimension at the specified position.
-Allocates an IBuffer of the given size on the GPU, using cudaMallocAsync.
Allocates an ITensor of the given dimensions on the GPU, using cudaMallocAsync.
Allocates an IBuffer of the given size and memory type.
Allocates an ITensor of the given dimensions and memory type.
Create an empty IBuffer of the given memory type. It may be resized later.
Create an empty ITensor of the given memory type. It may be reshaped later.
Set the contents of the given buffer to value.
Copy src to dst.
Copy src to dst.
Copy src into a new IBuffer with a potentially different memory type.
Copy src into a new ITensor with a potentially different memory type.
Copy src into a new IBuffer with a potentially different memory type.
Copy src into a new ITensor with a potentially different memory type.
Copy src into a new ITensor with a potentially different memory type.
Get the underlying cuda stream.
+The current size of the memory reserved by the memory pool.
+The current size of the memory used by the memory pool.
+The current size of the memory free in the memory pool.
+Try to trim the memory reserved by the pool to size bytes. This synchronizes implicitly with the stream.
Public Static Functions
Returns the volume of the dimensions. Returns -1 if d.nbDims < 0.
Returns the volume of the dimensions. Throws if d.nbDims < 0.
Returns the strides of each dimemsion in a Shape.
-Removes the given unit dimension from shape.
shape – The shape to squeeze.
dim – The dimension that should be removed (“squeezed”).
A new shape without the unit dimension.
-Add a unit dimension to shape at the specified position.
shape – The shape to unsqueeze.
dim – The dimension where unit dimension should be added.
A new shape with the added unit dimension.
-Creates a sliced view on the underlying tensor. The view will have the same data type as tensor.
tensor – The tensor to view.
offset – The offset of the view w.r.t. dimension 0 of the tensor.
size – The size of the view w.r.t. dimension 0 of the tensor.
A view on the buffer.
Allocates an IBuffer of the given size on the GPU, using cudaMalloc.
Allocates an ITensor of the given dimensions on the GPU, using cudaMalloc.
offsetDims – The offset in multiple dimensions.
tensor – The tensor to view.
offsetDims – The offset dimensions of the view.
size – The size of the view w.r.t. the last dimension in offsetDims.
offsetDims – specifies all dimensions.
Whenever – offset overflows or the last dimension offset+size overflows.
-A view of shape [size, the rest dimensions] or [size] when
-Allocates an IBuffer of the given size on the CPU.
Allocates an ITensor of the given dimensions on the CPU.
Allocates a pinned IBuffer of the given size on the CPU.
Allocates a pinned ITensor of the given dimensions on the CPU.
return the rest slices at the last dimension when size omitted.
Allocates a pinned IBuffer of the given size on the CPU in the default memory pool.
offsetDims – specifies all dimensions.
-Just the block at the point, with shape of [the rest dimensions] or [1] when
-Allocates a pinned ITensor of the given dimensions on the CPU in the default memory pool.
Returns a view on the underlying buffer (or tensor) with the given shape.
tensor – The tensor to view.
shape – The shape of the view.
A view on the tensor.
Allocates an IBuffer of the given size in UVM.
Returns a view on the underlying tensor which can be independently reshaped.
tensor – The tensor to view.
-A view on the tensor.
Allocates an ITensor of the given dimensions in UVM.
Returns a flattened view on the underlying tensor which can be independently reshaped.
tensor – The tensor to flatten.
sliceN – Slice the first N elements after flattening. -1 means take the whole flattened tensor.
A flatten view on the tensor.
Wraps the given data in an ITensor. The ITensor will not own the underlying data and cannot be reshaped beyond capacity.
data – The data to wrap.
type – The data type of the data.
shape – The shape of the tensor.
capacity – The capacity of the buffer.
An ITensor.
A convenience function to create a tensor shape with the given dimensions.
-A convenience function for converting a tensor shape to a string.
Protected Functions
-Protected Static Functions
-Private Members
+Friends
GPT decoder class with support for in-flight batching.
-Public Types
-Public Functions
-Setup the decoder before calling forward()
Disable Lookahead decoding.
-Run one step for all requests without blocking the host process and return the token for synchronization.
-Run one step for all requests and wait for completion on the host.
-Gather final beam search results for request batchSlot. Result will only be available after event returned.
Private Types
-Private Functions
-Calls decoders for tokens per engine step.
-Private Members
-Public Functions
Public Static Functions
+Private Members
Public Functions
+Public Members
+Public Types
+ + +Public Functions
+Private Members
+ + +Functions
+Helper function to produce batch slots [0, 1, …, batchSize - 1] for paths that do not explicitly provide batch slots to the decoder.
+Public Types
+Public Functions
+explicitDraftTokensDType – is only used by ExplicitDraftTokens model to WAR the lack of bf16 decoder.
+Private Members
+Subclassed by tensorrt_llm::runtime::GptDecoder< T >
+Public Types
+ + +Public Functions
+explicitDraftTokensDType – is only used by ExplicitDraftTokens model to WAR the lack of bf16 decoder.
+Public Static Functions
+Public Types
+Public Functions
+Public Members
+ + +Private Functions
+Private Members
+Public Members
+ + + + + + + + + + + + + + +Public Functions
+Public Members
+ + + + + + + + +[maxBatchSize, maxDecodingTokens] or [numSequences, maxDecodingTokens]
+[maxBatchSize, maxDecodingDraftTokens] or [numSequences, maxDecodingDraftTokens]
+[maxBatchSize, maxNumPaths, maxPathLen] or [numSequences, maxNumPaths, maxPathLen]
+[maxBatchSize, maxNumPaths, maxPathLen] or [numSequences, maxNumPaths, maxPathLen]
+[maxBatchSize, maxDecodingTokens, ceil(maxDecodingTokens / 32)] or [numGenSequences, maxDecodingTokens, ceil(maxDecodingTokens / 32)]
+[maxBatchSize * maxDecodingTokens] or [numSequences * maxDecodingTokens]
+[maxBatchSize, maxDecodingDraftTokens] or [numSequences, maxDecodingDraftTokens]
+[maxBatchSize, maxDecodingDraftTokens] or [numSequences, maxDecodingDraftTokens]
+[maxBatchSize, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens] or [numSequences, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens]
+Public Types
+ + +Public Functions
+Private Members
+Public Types
+Public Functions
+Public Members
+Private Functions
+Public Members
+ + + + + + + + + + + + + + + + + + + + + + + + +Subclassed by tensorrt_llm::runtime::ExplicitDraftTokensBuffers::EngineInputs
+Public Functions
+Public Members
+ + + + + + + + +[maxBatchSize, maxNumPaths, maxPathDraftLen] or [numGenSequences, maxNumPaths, maxPathDraftLen]
+[maxBatchSize, maxNumPaths, maxPathLen] or [numGenSequences, maxNumPaths, maxPathLen]
+[maxBatchSize, maxNumPaths, maxPathLen] or [numGenSequences, maxNumPaths, maxPathLen]
+[maxBatchSize, maxNumPaths, maxPathDraftLen, vocabSize] or [numGenSequences, maxNumPaths, maxPathDraftLen, vocabSize]
+Functions
+Utility function to print a shape.
+Utility function to print a tensor with its shape.
+Retrieves a T const typed pointer to the underlying data of the tensor pointed to by the tensorPtr, or nullptr if the tensorPtr is null.
+This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
+T – The type of the underlying data.
+tensorPtr – A possibly null shared ptr.
+A pointer to T const, possibly nullptr.
+Retrieves a T typed pointer to the underlying data of the buffer pointed to by the tensorPtr, or nullptr if the tensorPtr is null.
+This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
+T – The type of the underlying data.
+tensorPtr – A possibly null shared ptr.
+A pointer to T, possibly nullptr.
+Retrieves a T typed pointer to the underlying data of the tensor pointed to by the tensor pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
+This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
+T – The type of the underlying data.
+optionalBufferPtr – A possibly empty optional.
+A pointer to T, possibly nullptr.
+Retrieves a T const typed pointer to the underlying data of the tensor pointed to by the tensor pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
+This overload has to be declared to avoid ambiguity when an implicit conversion to IBuffer is involved.
+T – The type of the underlying data.
+optionalBufferPtr – A possibly empty optional.
+A pointer to const T, possibly nullptr.
+Public Types
+ + + + + + + + + + + + +Public Functions
+Returns the tensor n-th dimension. If n is negative, returns the (nbDims - n)th dimension. TODO: replace with constexpr parameter when moving to C++20.
+Sets the tensor dimensions. The new size of the tensor will be volume(dims)
Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity.
+Removes the given unit dimensions from this tensor.
+Adds a unit dimension at the specified position.
+Public Static Functions
+Returns the volume of the dimensions. Returns -1 if d.nbDims < 0.
Returns the volume of the dimensions. Throws if d.nbDims < 0.
Returns the strides of each dimemsion in a Shape.
+Removes the given unit dimension from shape.
shape – The shape to squeeze.
dim – The dimension that should be removed (“squeezed”).
A new shape without the unit dimension.
+Add a unit dimension to shape at the specified position.
shape – The shape to unsqueeze.
dim – The dimension where unit dimension should be added.
A new shape with the added unit dimension.
+Creates a sliced view on the underlying tensor. The view will have the same data type as tensor.
tensor – The tensor to view.
offset – The offset of the view w.r.t. dimension 0 of the tensor.
size – The size of the view w.r.t. dimension 0 of the tensor.
A view on the buffer.
offsetDims – The offset in multiple dimensions.
tensor – The tensor to view.
offsetDims – The offset dimensions of the view.
size – The size of the view w.r.t. the last dimension in offsetDims.
offsetDims – specifies all dimensions.
Whenever – offset overflows or the last dimension offset+size overflows.
+A view of shape [size, the rest dimensions] or [size] when
+return the rest slices at the last dimension when size omitted.
offsetDims – specifies all dimensions.
+Just the block at the point, with shape of [the rest dimensions] or [1] when
+Returns a view on the underlying buffer (or tensor) with the given shape.
tensor – The tensor to view.
shape – The shape of the view.
A view on the tensor.
Returns a view on the underlying tensor which can be independently reshaped.
tensor – The tensor to view.
+A view on the tensor.
Returns a flattened view on the underlying tensor which can be independently reshaped.
tensor – The tensor to flatten.
sliceN – Slice the first N elements after flattening. -1 means take the whole flattened tensor.
A flatten view on the tensor.
Wraps the given data in an ITensor. The ITensor will not own the underlying data and cannot be reshaped beyond capacity.
data – The data to wrap.
type – The data type of the data.
shape – The shape of the tensor.
capacity – The capacity of the buffer.
An ITensor.
A convenience function to create a tensor shape with the given dimensions.
+A convenience function for converting a tensor shape to a string.
A convenience function to compare shapes.
+A convenience function to compare shapes.
+Protected Functions
+Friends
+Defines
+Typedefs
+Enums
+ + +Public Functions
+Public Functions
+Is my rank the last rank in its pipeline?
+Public Static Functions
+Public Static Attributes
+Private Members
+Functions
+Public Types
+Values:
+Public Functions
+Public Static Functions
+Private Members
+Public Types
+Public Functions
+Public Static Functions
+Private Functions
+Private Members
+Private Static Attributes
+Public Types
+Public Functions
+Creates a new cuda event. The event will be destroyed in the destructor.
+flags – Flags for event creation. By default, event timing is disabled.
+Pass an existing cuda event to this object.
+event – The event to pass to this object.
ownsEvent – Whether this object owns the event and destroys it in the destructor.
Synchronizes the event.
+Private Types
+ + +Represents the inputs to the decoder.
+This input type is assumed immutable. It represents whatever the decoder received initially, and can always be referred to as such.
+Public Types
+Public Functions
+Public Members
+Mandatory parameters The index of the decoding step we are on. Only used in Python runtime
+The maximum number of tokens to decode.
+The maximum length of the attention window to consider while decoding.
+The number of tokens to use as attention sinks, https://arxiv.org/html/2309.17453v3.
+The number of samples in the batch.
+The beam widths of each request, [batchSize].
+The maximum value in the stopWordsLens tensor.
The maximum value in the badWordsLens tensor.
The output of the model forward computation, a probability distribution over the vocabulary [batchSize][numGenTokens, beamWidth, vocabSizePadded] on gpu
+The end ids, [batchSize * beamWidth] on gpu.
+Address map of the linear batch id to to the seq slots, [batchSize] on pinned, int32_t.
+Optional parameters Finished states at current iteration (skip decoding step of a request if true), [batchSize, beamWidth] on gpu
+The maximum sequence length for each sequence in the batch, [batchSize] on gpu.
+Parameters for beam search KV cache index for beam search, [batchSize, beamWidth, maxSeqLen] on gpu
+Steps of each request, for Variable-Beam-Width-Search, [batchSize].
+Public Members
+Public Members
+Public Members
+ + + + + + + + + + + + + + + + +Public Members
+[batchSize, maxTokensPerStep, maxMedusaHeads + 1], on gpu
+[batchSize, maxTokensPerStep], on gpu
+[batchSize][maxAcceptedDraftTokensPerStep][maxDraftTokens + 1, vocabSizePadded], on gpu
+[batchSize], on gpu
+GPT decoder class with support for in-flight batching.
+Subclassed by tensorrt_llm::runtime::GptDecoderBatched
+Public Types
Public Functions
Public Types
- - -Public Functions
-Setup the decoder before calling forward()
Disable Lookahead decoding.
+Run one step for all requests without blocking the host process and return the token for synchronization.
+Run one step for all requests and wait for completion on the host.
+Public Members
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Public Types
- - -Public Functions
-Public Types
- - -Public Functions
-Public Types
- - -Public Functions
-Private Members
-Typedefs
- - -Enums
- - -Functions
-Gets a typed pointer to the constant underlying data of the buffer.
-T – The type of the underlying data.
-buffer – The buffer to get a pointer to.
-A pointer to constant T.
Gets a typed pointer to the underlying data of the buffer.
-T – The type of the underlying data.
-buffer – The buffer to get a pointer to.
-A pointer to T.
Retrieves a T typed pointer to the underlying data of the buffer pointed to by the bufferPtr, or nullptr if the bufferPtr is null.
-T – The type of the underlying data.
-bufferPtr – A possibly null shared ptr.
-A pointer to T, possibly nullptr.
-Retrieves a T const typed pointer to the underlying data of the buffer pointed to by the bufferPtr, or nullptr if the bufferPtr is null.
-T – The type of the underlying data.
-bufferPtr – A possibly null shared ptr.
-A pointer to const T, possibly nullptr.
-Retrieves a T typed pointer to the underlying data of the buffer pointed to by the buffer pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
-T – The type of the underlying data.
-optionalBufferPtr – A possibly empty optional.
-A pointer to T, possibly nullptr.
-Retrieves a T const typed pointer to the underlying data of the buffer pointed to by the buffer pointer contained in the optionalBufferPtr, or nullptr if the optional doesn’t have a value.
-T – The type of the underlying data.
-optionalBufferPtr – A possibly empty optional.
-A pointer to const T, possibly nullptr.
-A wrapper around nvinfer1::DataType that provides a support for pointer types.
Public Functions
-Public Static Attributes
- - -Public Types
-For converting a TensorRT data type to a C++ data type.
-Public Types
-Public Types
-Public Types
-Public Types
-Public Types
-Public Types
-Public Types
-Public Types
-Public Types
-Subclassed by tensorrt_llm::runtime::ITensor
-Public Types
- - - - - - - - - - -Public Functions
-Returns a pointer to underlying array.
-Returns a pointer to underlying array.
-Returns a pointer to the underlying array at a given element index.
-Returns a pointer to the underlying array at a given element index.
-Returns the size (in number of elements) of the buffer.
-Returns the size (in bytes) of the buffer.
-Returns the capacity of the buffer.
-Returns the memory type of the buffer.
-Resizes the buffer. This is a no-op if the new size is smaller than or equal to the current capacity.
-Releases the buffer. It will be reset to nullptr.
-Public Static Functions
- - -Creates a sliced view on the underlying buffer. The view will have the same data type as buffer.
buffer – The buffer to view.
offset – The offset of the view.
size – The size of the view.
A view on the buffer.
Returns a view on the underlying tensor which can be independently resized.
tensor – The tensor to view.
-A view on the tensor.
Returns a view on the underlying tensor with a different size.
tensor – The tensor to view.
size – The size of the view.
A view on the tensor.
Wraps the given data in an IBuffer. The IBuffer will not own the underlying data and cannot be resized beyond capacity.
data – The data to wrap.
type – The data type of the data.
size – The size of the buffer.
capacity – The capacity of the buffer.
An IBuffer.
Determine the memory type of a pointer.
+) const = 0#Gather final beam search results for request batchIdx. Result will only be available after event returned.
Public Static Attributes
-Public Types
+Public Static Attributes
-Public Static Attributes
-For converting a C++ data type to a TensorRT data type.
-Public Static Attributes
-Public Static Attributes
-Public Static Attributes
-Public Static Attributes
-Public Static Attributes
-Public Static Attributes
-Private Static Attributes
+Public Functions
+Public Members
[maxDecoderSteps][batchSize][1, beamWidth, vocabSizePadded], on gpu
+Mandatory parameters Logits
+Maximum number of decoding tokens of active slots.
+Public Functions
+Private Members
+GPT decoder class with support for in-flight batching.
+Public Types
+Public Functions
+Setup the decoder before calling forward()
Disable Lookahead decoding.
+Run one step for all requests without blocking the host process and return the token for synchronization.
+Run one step for all requests and wait for completion on the host.
+Gather final beam search results for request batchSlot. Result will only be available after event returned.
Private Types
+Private Functions
+Calls decoders for tokens per engine step.
+Private Members
+Public Functions
+Creates a new cuda stream on the current device. The stream will be destroyed in the destructor.
+flags – Flags for stream creation. See cudaStreamCreateWithFlags for a list of valid flags that can be passed.
priority – Priority of the stream. Lower numbers represent higher priorities. See cudaDeviceGetStreamPriorityRange for more information about the meaningful stream priorities that can be passed.
Pass an existing cuda stream to this object.
+stream – The stream to pass to this object.
device – The device on which the stream was created.
ownsStream – Whether this object owns the stream and destroys it in the destructor.
Construct with an existing cuda stream or the default stream by passing nullptr.
+Returns the device on which the stream was created.
+Returns the stream associated with this object.
+Synchronizes the stream.
+Private Types
+ + +Functions
+Public Members
+Defines
+Public Functions
+Public Members
+Private Types
+Private Functions
+ + +Public Types
+Public Functions
+Public Members
+Mandatory parameters.
+Public Functions
+Public Members
+Public Types
+ + +Public Functions
+Setup buffers for the decoder excluding speculative decoding.
+Setup buffers for the cache indirection.
+This is used for beam search on pipeline parallel ranks without a decoder.
+Setup buffers for speculative decoding.
+Disable lookahead decoding.
+[batchSize], number of finished sequences per request, on gpu
+[batchSize, beamWidth], FinishedState value, on gpu
+[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding, on gpu. In case of beam search, contains the ungathered data.
+batchIdx – index of the batch
+[maxBeamWidth, maxInputLength + maxNewTokens], contains input token ids and generated token ids without padding for request batchIdx, on gpu. In case of beam search, contains the ungathered data.
[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], only used for beam search. It contains gathered token ids without padding, on gpu.
+batchIdx – index of the batch
+[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], only used for beam search. It contains gathered token ids without padding for request batchIdx, on gpu.
[batchSize, maxBeamWidth, maxInputLength + maxNewTokens], contains parent ids collected during beam search without padding, on gpu
+[batchSize, maxBeamWidth], cumulative log probabilities (per beam), on gpu
+[maxBeamWidth], cumulative log probabilities (per beam), on gpu
+[batchSize, maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu
+[maxBeamWidth, maxSequenceLength], log probabilities (per beam), on gpu
+[batchSize, maxBeamWidth], sequence lengths, on gpu
+batchIdx – index of the batch
+[maxBeamWidth], sequence lengths for request batchIdx, on gpu
Get maxTokensPerStep tokens generated in the last forward pass.
+[maxTokensPerStep, batchSize, maxBeamWidth], tokens generated in last forward pass, on gpu
+[batchSize, maxDraftTokens], predicted draft tokens for next step, on gpu
+[batchSize], predicted draft tokens lengths for previous step, on gpu
+[batchSize], predicted draft tokens lengths for next step, on gpu
+[batchSize + 1], exclusive sum of accepted draft token lengths, on gpu
+[batchSize, maxAcceptedDraftTokensPerStep], accepted paths packed into continuous tensor, on gpu
+[maxTokensPerStep, batchSize, beamWidth], finished states of type FinishedState, on gpu
+Get the number of tokens for all requests in the batch.
+The number of tokens for all requests in the batch.
+Get the number of tokens for a specific request in the batch.
+batchIdx – The index of the request in the batch.
+The number of tokens for the specified request.
+Set the number of tokens for a specific request in the batch.
+batchIdx – The index of the request in the batch.
numTokens – The number of tokens for the specified request.
Get the speculative decoding mode.
+Get the explicit draft tokens buffers.
+Get the eagle buffers.
+Get the lookahead buffers.
+Workspace for beam search in streaming mode.
+Get the generation steps for all requests in the batch.
+The generation steps for all requests in the batch.
+Set the generation steps for all requests in the batch.
+generationSteps – The generation steps for all requests in the batch.
+Stateful inputs for the decoder. Allocated for maxBatchSize slots.
+Stateful outputs for the decoder. Allocated for maxBatchSize slots.
+Private Functions
+Private Members
+Stateful inputs for the decoder. Allocated for maxBatchSize slots.
+Stateful outputs for the decoder. Allocated for maxBatchSize slots.
+[maxTokensPerStep, batchSize, beamWidth] finished states of type FinishedState for each generated token of maxTokensPerStep, on gpu
+Workspace for beam search in streaming mode.
+[batchSize], the num tokens of each request.
+Functions
+Public Functions
+Public Functions
+Public Static Attributes
Private Functions
+Public Functions
+Public Static Functions
+Private Members
+