kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
21aff2e313
TensorRT-LLMs/cpp/tensorrt_llm/runtime
History
dongxuy04 21aff2e313
feat: large-scale EP(part 2: MoE Load Balancer - core utilities) (#4384) 
* first commit of cpp moe loadbalance code

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add python bindings for moe load balance

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add python wrapper, ut and bug fixes

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add binding for layerId and update binding test

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

* add host tensor sharing and ut

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>

---------

Signed-off-by: Dongxu Yang <78518666+dongxuy04@users.noreply.github.com>
2025-05-20 17:53:48 +08:00
..
utils [TRTLLM-5171] chore: Remove GptSession/V1 from TRT workflow (#4092) 2025-05-14 23:10:04 +02:00
bufferManager.cpp Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
bufferView.h Update TensorRT-LLM (#2783) 2025-02-13 18:40:22 +08:00
CMakeLists.txt feat: large-scale EP(part 2: MoE Load Balancer - core utilities) (#4384) 2025-05-20 17:53:48 +08:00
cudaMemPool.cpp Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
cudaMemPool.h Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
decoderState.cpp [TRTLLM-5171] chore: Remove GptSession/V1 from TRT workflow (#4092) 2025-05-14 23:10:04 +02:00
decodingLayerWorkspace.cpp Update TensorRT-LLM (#2184) 2024-09-03 12:14:23 +02:00
decodingLayerWorkspace.h Update TensorRT-LLM (#2436) 2024-11-12 15:27:49 +08:00
decodingOutput.cpp Update (#2978) 2025-03-23 16:39:35 +08:00
eagleBuffers.cpp fix: Eagle decoding in TRT flow (#4229) 2025-05-14 16:10:49 +02:00
explicitDraftTokensBuffers.cpp Update TensorRT-LLM (#2873) 2025-03-11 21:13:42 +08:00
explicitDraftTokensModule.h Update TensorRT-LLM (#1763) 2024-06-11 16:59:02 +08:00
gptDecoder.cpp Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
gptDecoderBatched.cpp Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
gptJsonConfig.cpp Feat: Variable-Beam-Width-Search (VBWS) part3 (#3338) 2025-04-08 23:51:27 +08:00
iBuffer.cpp Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
ipcNvlsMemory.cpp refactor: Introduce MpiTag enumeration and update MPI function signatures (#3893) 2025-05-04 13:24:29 +02:00
ipcSocket.cpp Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
ipcSocket.h Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
ipcUtils.cpp chore: remove usernames from comments (#3291) 2025-04-05 13:44:28 +08:00
iTensor.cpp Update TensorRT-LLM (#2849) 2025-03-04 18:44:00 +08:00
jsonSerialization.h Update TensorRT-LLM (#2436) 2024-11-12 15:27:49 +08:00
layerProfiler.cpp Update TensorRT-LLM (#1554) 2024-05-07 23:34:28 +08:00
layerProfiler.h Update TensorRT-LLM (#1554) 2024-05-07 23:34:28 +08:00
lookaheadBuffers.cpp Feat: Variable-Beam-Width-Search (VBWS) part3 (#3338) 2025-04-08 23:51:27 +08:00
loraCache.cpp fix: [nvbugs/5287097] Align PP layer distribution between pytorch and TRT flow. (#4399) 2025-05-19 14:25:36 -07:00
loraManager.cpp fix: [nvbugs/5287097] Align PP layer distribution between pytorch and TRT flow. (#4399) 2025-05-19 14:25:36 -07:00
loraManager.h Update TensorRT-LLM (#2413) 2024-11-05 16:27:06 +08:00
loraModule.cpp Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
loraUtils.cpp Update TensorRT-LLM (#2820) 2025-02-25 21:21:49 +08:00
loraUtils.h Update TensorRT-LLM (#2820) 2025-02-25 21:21:49 +08:00
memoryCounters.cpp Update TensorRT-LLM (#2110) 2024-08-13 22:34:33 +08:00
moeLoadBalancer.cpp feat: large-scale EP(part 2: MoE Load Balancer - core utilities) (#4384) 2025-05-20 17:53:48 +08:00
moeLoadBalancer.h feat: large-scale EP(part 2: MoE Load Balancer - core utilities) (#4384) 2025-05-20 17:53:48 +08:00
ncclCommunicator.cpp Update TensorRT-LLM (#2755) 2025-02-11 03:01:00 +00:00
ncclCommunicator.h Update TensorRT-LLM (#2792) 2025-02-18 21:27:39 +08:00
promptTuningParams.cpp Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
runtimeKernels.cu [TRTLLM-5171] chore: Remove GptSession/V1 from TRT workflow (#4092) 2025-05-14 23:10:04 +02:00
runtimeKernels.h [TRTLLM-5171] chore: Remove GptSession/V1 from TRT workflow (#4092) 2025-05-14 23:10:04 +02:00
tensorView.h Update TensorRT-LLM (#1793) 2024-06-18 18:18:23 +08:00
tllmBuffers.cpp Update TensorRT-LLM (#2792) 2025-02-18 21:27:39 +08:00
tllmBuffers.h chore: Stabilize ABI boundary for internal kernel library (#3117) 2025-04-11 15:07:50 +08:00
tllmLogger.cpp Update TensorRT-LLM (#787) 2024-01-02 17:54:32 +08:00
tllmRuntime.cpp Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
tllmRuntime.h Feat: Variable-Beam-Width-Search (VBWS) part4 (#3979) 2025-05-12 22:32:29 +02:00
tllmStreamReaders.cpp feat: Integrate GPUDirect Storage (GDS) into Executor API (#3582) 2025-04-18 15:59:21 +08:00
tllmStreamReaders.h feat: Integrate GPUDirect Storage (GDS) into Executor API (#3582) 2025-04-18 15:59:21 +08:00
torch.h Update TensorRT-LLM (#2110) 2024-08-13 22:34:33 +08:00
torchUtils.h [TRTLLM-3330][feat] Support DeepSeek-R1 W4A8 on Hopper (#4123) 2025-05-14 15:48:07 +08:00
torchView.h Update TensorRT-LLM (#1168) 2024-02-27 17:37:34 +08:00
workerPool.cpp Update TensorRT-LLM (#2156) 2024-08-27 18:20:59 +08:00
workerPool.h Update TensorRT-LLM (#2156) 2024-08-27 18:20:59 +08:00
worldConfig.cpp bug: Fix hang bug when context server doesn't have enough capacity for KV Cache (#3095) 2025-04-21 15:16:55 +08:00