mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-14 06:27:45 +08:00
8340657ae4
* refactor: Restructure DecoderBuffers and DecoderStepAsyncSend - Move communication logic from `DecoderBuffers` to `DecoderStepAsyncSend`. - Updated `DecoderStepAsyncSend` constructor to utilize the `DecoderBuffers`, enhancing clarity and reducing parameter complexity. - Refactored related methods to align with the new class structure, improving maintainability and readability of the code. These changes streamline the handling of decoding buffers and improve the overall architecture of the batch manager. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Restructure SlotDecoderBuffers and DecoderSlotAsyncSend - Move communication logic from `SlotDecoderBuffers` to `DecoderSlotAsyncSend`. - Updated `DecoderSlotAsyncSend` constructor to utilize the `SlotDecoderBuffers`, enhancing clarity and reducing parameter complexity. - Refactored related methods to align with the new class structure, improving maintainability and readability of the code. These changes enhance the structure and readability of the batch manager's decoding process. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * chore: Log DecodingMode Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Introduce DecoderOutputBuffers and update related classes - Moved buffers from `DecoderBuffers` to `DecoderOutputBuffers` to better reflect its purpose. - Updated the `DecoderStepAsyncSend` class to utilize `DecoderOutputBuffers`, enhancing clarity in the communication logic. - Refactored the constructor and methods in `DecoderBuffers` to accommodate the new structure, improving maintainability. - Added Python bindings for `DecoderOutputBuffers` to ensure compatibility with existing interfaces. These changes streamline the handling of output buffers in the decoding process, improving the overall architecture of the batch manager. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Update MPI communicator handling - Changed the `commSession` parameter type from `std::shared_ptr<mpi::MpiComm>` to `mpi::MpiComm` in `DecoderStepAsyncSend` and `DecoderSlotAsyncSend` classes for improved clarity and reduced complexity. - Updated related methods and constructors to reflect the new parameter type, enhancing maintainability. - Refactored the `TrtGptModelInflightBatching` class to accommodate these changes, ensuring consistent usage of `MpiComm`. These modifications streamline the communication logic in the decoding process, improving the overall architecture of the batch manager. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> * refactor: Replace shared_ptr with unique_ptr for buffer management - Updated the `TrtGptModelInflightBatching` class to use `std::unique_ptr` instead of `std::shared_ptr` for various buffer types, including `AllReduceBuffers`, `RuntimeBuffers`, `DecoderBuffers`, and `SlotDecoderBuffers`. - This change enhances memory management and ownership semantics, reducing overhead and improving performance. These modifications contribute to a cleaner and more efficient architecture in the batch manager. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> --------- Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| utils | ||
| allocateKvCache.cpp | ||
| assignReqSeqSlots.cpp | ||
| cacheFormatter.cpp | ||
| cacheFormatter.h | ||
| cacheTransceiver.cpp | ||
| capacityScheduler.cpp | ||
| CMakeLists.txt | ||
| contextProgress.cpp | ||
| createNewDecoderRequests.cpp | ||
| dataTransceiver.cpp | ||
| dataTransceiver.h | ||
| dataTransceiverImpl.cpp | ||
| dataTransceiverImpl.h | ||
| decoderBuffers.cpp | ||
| encoderBuffers.cpp | ||
| encoderBuffers.h | ||
| evictionPolicy.cpp | ||
| generateRequestOptions.cpp | ||
| guidedDecoder.cpp | ||
| handleContextLogits.cpp | ||
| handleGenerationLogits.cpp | ||
| kvCacheEventManager.cpp | ||
| kvCacheManager.cpp | ||
| kvCacheTransferManager.cpp | ||
| llmRequest.cpp | ||
| logitsPostProcessor.cpp | ||
| loraBuffers.cpp | ||
| loraBuffers.h | ||
| makeDecodingBatchInputOutput.cpp | ||
| medusaBuffers.cpp | ||
| microBatchScheduler.cpp | ||
| mlaCacheFormatter.cpp | ||
| mlaCacheFormatter.h | ||
| pauseRequests.cpp | ||
| peftCacheManager.cpp | ||
| promptTuningBuffers.cpp | ||
| rnnStateBuffers.cpp | ||
| rnnStateBuffers.h | ||
| rnnStateManager.cpp | ||
| runtimeBuffers.cpp | ||
| sequenceSlotManager.cpp | ||
| transformerBuffers.cpp | ||
| trtEncoderModel.cpp | ||
| trtEncoderModel.h | ||
| trtGptModel.h | ||
| trtGptModelFactory.h | ||
| trtGptModelInflightBatching.cpp | ||
| trtGptModelInflightBatching.h | ||
| trtGptModelV1.cpp | ||
| trtGptModelV1.h | ||