mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-14 06:27:45 +08:00
72057a0a64
* disable overlap in encoder Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * feat: invokeGatherBatch Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * feat: overlap same batch Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * chore: add enableTrtOverlap to ExecutorConfig Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * disable overlap for beam search and spec decode Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * skip overlap tests with beam search or speculative decoding Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * moveFinishedContextRequestsToGeneration and skip unfinished requests in updateRequests Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * enable overlap in GptChunkedLongContextTests Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * feat: Enable overlap in gptManagerBenchmark Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * feat: Improve early exit Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Use OptionalRef for newOutputTokens tensor Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * feat: Add overlap scheduling support to TRTLLMDecoder - Updated TRTLLMDecoder to accept an `enable_overlap_scheduler` parameter. - Modified the decoder's internal logic to utilize the overlap scheduling feature. - Adjusted the sequence lengths handling to ensure compatibility with the new scheduling approach. - Enhanced unit tests to include cases for the overlap scheduler with the TRTLLMDecoder. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * fix: allNewTokens in PP Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> --------- Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| utils | ||
| bufferManager.cpp | ||
| bufferView.h | ||
| CMakeLists.txt | ||
| cudaMemPool.cpp | ||
| cudaMemPool.h | ||
| decoderState.cpp | ||
| decodingLayerWorkspace.cpp | ||
| decodingLayerWorkspace.h | ||
| decodingOutput.cpp | ||
| eagleBuffers.cpp | ||
| explicitDraftTokensBuffers.cpp | ||
| explicitDraftTokensModule.h | ||
| generationConfig.cpp | ||
| generationConfig.h | ||
| gptDecoder.cpp | ||
| gptDecoderBatched.cpp | ||
| gptJsonConfig.cpp | ||
| gptSession.cpp | ||
| iBuffer.cpp | ||
| ipcNvlsMemory.cpp | ||
| ipcSocket.cpp | ||
| ipcSocket.h | ||
| ipcUtils.cpp | ||
| iTensor.cpp | ||
| jsonSerialization.h | ||
| layerProfiler.cpp | ||
| layerProfiler.h | ||
| lookaheadBuffers.cpp | ||
| loraCache.cpp | ||
| loraManager.cpp | ||
| loraManager.h | ||
| loraModule.cpp | ||
| loraUtils.cpp | ||
| loraUtils.h | ||
| memoryCounters.cpp | ||
| ncclCommunicator.cpp | ||
| ncclCommunicator.h | ||
| promptTuningParams.cpp | ||
| rnnStateBuffers.cpp | ||
| rnnStateBuffers.h | ||
| runtimeBuffers.cpp | ||
| runtimeBuffers.h | ||
| runtimeKernels.cu | ||
| runtimeKernels.h | ||
| statefulGptDecoder.cpp | ||
| statefulGptDecoder.h | ||
| statefulGptDecoderBatched.cpp | ||
| tensorView.h | ||
| tllmBuffers.cpp | ||
| tllmBuffers.h | ||
| tllmLogger.cpp | ||
| tllmRuntime.cpp | ||
| tllmRuntime.h | ||
| tllmStreamReaders.cpp | ||
| tllmStreamReaders.h | ||
| torch.h | ||
| torchUtils.h | ||
| torchView.h | ||
| transformerBuffers.cpp | ||
| transformerBuffers.h | ||
| workerPool.cpp | ||
| workerPool.h | ||
| worldConfig.cpp | ||