mirror of
https://github.com/NVIDIA/TensorRT-LLM.git
synced 2026-01-22 03:35:00 +08:00
9e7182b603
Four distinct strategies are implemented to accommodate different distributed tuning scenarios, including BROADCAST, INDEPENDENT, MERGE, PARALLEL. * Distributed tuning is disabled by default, with the INDEPENDENT strategy as the fallback. This conservative approach prevents unexpected behavior in standard use cases. * Only operations with significant tuning time overhead have been assigned the PARALLEL strategy, which allows the same tensor parallelism (TP) rank to tune tactics concurrently across different ranks. This targeted approach balances performance gains with stability. * Operations with nested tuning structures, such as NVFP4GemmUnifiedRunner, currently support only the INDEPENDENT strategy. This restriction exists because the synchronization mechanism is optimized only for leaf operations and doesn't yet handle nested hierarchies. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _util.py | ||
| config_utils.py | ||
| cuda_graph_runner.py | ||
| executor_request_queue.py | ||
| finish_reason.py | ||
| grammar_matcher.py | ||
| guided_decoder.py | ||
| handle_additional_outputs.py | ||
| handle_logits.py | ||
| kv_cache_connector.py | ||
| kv_cache_transceiver.py | ||
| layerwise_nvtx_marker.py | ||
| llm_request.py | ||
| make_decoding_batch_input_output.py | ||
| mamba_cache_manager.py | ||
| model_engine.py | ||
| model_loader.py | ||
| py_executor_creator.py | ||
| py_executor.py | ||
| resource_manager.py | ||
| sampler.py | ||
| sampling_utils_flashinfer.py | ||
| sampling_utils.py | ||
| scheduler.py | ||
| seq_slot_manager.py | ||