kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

[https://nvbugs/5569754][chore] Adjust max batch size to prevent OOM (#8876)

Browse Source 
Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
This commit is contained in:
JunyiXu-nv 2025-11-05 01:34:26 +08:00 committed by GitHub
parent cacb8a84f2
commit c329f5f78b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
examples/llm-api/llm_mgmn_llm_distributed.sh
View File

@ -51,5 +51,6 @@ srun -l \
trtllm-llmapi-launch python3 $script \
--model_dir $LOCAL_MODEL \
--prompt 'Hello, how are you?' \
--tp_size 2
--tp_size 2 \
--max_batch_size 256
"