mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-14 06:27:45 +08:00
d7386d14a8
* refactor: Simplifiy disableLookahead method Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * Update DecoderBuffers comments Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Move numDecodingEngineTokens to DecoderState This commit introduces new methods in the DecoderState class to manage the number of tokens for each request in a batch. The following changes were made: - Added `getNumDecodingEngineTokens()` to retrieve the number of tokens for all requests. - Added `getNumDecodingEngineTokens(SizeType32 batchIdx)` to get the token count for a specific request. - Added `setNumDecodingEngineTokens(SizeType32 batchIdx, SizeType32 numTokens)` to set the token count for a specific request. - Updated the setup method to initialize the token count vector based on the maximum batch size. - Refactored the `CreateNewDecoderRequests` class to utilize the new token management methods, improving clarity and maintainability. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Improve shape variables in DecoderState Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> --------- Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| utils | ||
| bufferManager.cpp | ||
| bufferView.h | ||
| CMakeLists.txt | ||
| cudaMemPool.cpp | ||
| cudaMemPool.h | ||
| decoderState.cpp | ||
| decodingLayerWorkspace.cpp | ||
| decodingLayerWorkspace.h | ||
| decodingOutput.cpp | ||
| eagleBuffers.cpp | ||
| explicitDraftTokensBuffers.cpp | ||
| explicitDraftTokensModule.h | ||
| generationConfig.cpp | ||
| generationConfig.h | ||
| gptDecoder.cpp | ||
| gptDecoderBatched.cpp | ||
| gptJsonConfig.cpp | ||
| gptSession.cpp | ||
| iBuffer.cpp | ||
| ipcNvlsMemory.cpp | ||
| ipcSocket.cpp | ||
| ipcSocket.h | ||
| ipcUtils.cpp | ||
| iTensor.cpp | ||
| jsonSerialization.h | ||
| layerProfiler.cpp | ||
| layerProfiler.h | ||
| lookaheadBuffers.cpp | ||
| loraCache.cpp | ||
| loraManager.cpp | ||
| loraManager.h | ||
| loraModule.cpp | ||
| loraUtils.cpp | ||
| loraUtils.h | ||
| memoryCounters.cpp | ||
| ncclCommunicator.cpp | ||
| ncclCommunicator.h | ||
| promptTuningParams.cpp | ||
| rnnStateBuffers.cpp | ||
| rnnStateBuffers.h | ||
| runtimeBuffers.cpp | ||
| runtimeBuffers.h | ||
| runtimeKernels.cu | ||
| runtimeKernels.h | ||
| statefulGptDecoder.cpp | ||
| statefulGptDecoder.h | ||
| statefulGptDecoderBatched.cpp | ||
| tensorView.h | ||
| tllmBuffers.cpp | ||
| tllmBuffers.h | ||
| tllmLogger.cpp | ||
| tllmRuntime.cpp | ||
| tllmRuntime.h | ||
| torch.h | ||
| torchUtils.h | ||
| torchView.h | ||
| transformerBuffers.cpp | ||
| transformerBuffers.h | ||
| workerPool.cpp | ||
| workerPool.h | ||
| worldConfig.cpp | ||