kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
f11aeed624
TensorRT-LLMs/_cpp_gen/executor.html
Kaiyu Xie f11aeed624
Update gh-pages (#2651)
2025-01-03 15:12:39 +08:00

8526 lines
1.9 MiB
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Executor &mdash; tensorrt_llm documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=80d5e7a1" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=e59714d7" />
<link rel="stylesheet" type="text/css" href="../_static/copybutton.css?v=76b2166b" />
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9a2dae69"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/clipboard.min.js?v=a7894cd8"></script>
<script src="../_static/copybutton.js?v=65e89d2a"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Runtime" href="runtime.html" />
<link rel="prev" title="Runtime" href="../python-api/tensorrt_llm.runtime.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
tensorrt_llm
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Getting Started</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../quick-start-guide.html">Quick Start Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../key-features.html">Key Features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">Release Notes</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Installation</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../installation/linux.html">Installing on Linux</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/build-from-source-linux.html">Building from Source Code on Linux</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/windows.html">Installing on Windows</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/build-from-source-windows.html">Building from Source Code on Windows</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/grace-hopper.html">Installing on Grace Hopper</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">LLM API</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../llm-api/index.html">API Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="../llm-api/reference.html">API Reference</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">LLM API Examples</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../llm-api-examples/index.html">LLM Examples Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="../llm-api-examples/customization.html">Common Customizations</a></li>
<li class="toctree-l1"><a class="reference internal" href="../llm-api-examples/llm_api_examples.html">Examples</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Model Definition API</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.layers.html">Layers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.functional.html">Functionals</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.models.html">Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.plugin.html">Plugin</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.quantization.html">Quantization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../python-api/tensorrt_llm.runtime.html">Runtime</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">C++ API</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Executor</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#disaggserverutil-h">disaggServerUtil.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv412tensorrt_llm"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#executor-h">executor.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::batch_manager</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::batch_manager::kv_cache_manager</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RetentionPriority</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventData</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7versionEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::version()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::RequestIdType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdTypePv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERK18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERR18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsaSERK18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsaSERR18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::~ContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18ContextPhaseParamseqERK18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NKR12tensorrt_llm8executor18ContextPhaseParams17getFirstGenTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::getFirstGenTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NO12tensorrt_llm8executor18ContextPhaseParams17popFirstGenTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::popFirstGenTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18ContextPhaseParams8getReqIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::getReqId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18ContextPhaseParams8getStateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::getState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8getStateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::getState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams12releaseStateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::releaseState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8StatePtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::StatePtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams6mReqIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::mReqId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams15mFirstGenTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::mFirstGenTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams6mStateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::mState</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextPhaseParams::deleter()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig11DebugConfigEbb9StringVec10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::DebugConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfigeqERK11DebugConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig20getDebugInputTensorsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::getDebugInputTensors()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig21getDebugOutputTensorsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::getDebugOutputTensors()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig19getDebugTensorNamesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::getDebugTensorNames()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig28getDebugTensorsMaxIterationsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::getDebugTensorsMaxIterations()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig20setDebugInputTensorsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::setDebugInputTensors()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig21setDebugOutputTensorsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::setDebugOutputTensors()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig19setDebugTensorNamesERK9StringVec"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::setDebugTensorNames()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig28setDebugTensorsMaxIterationsE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::setDebugTensorsMaxIterations()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::StringVec</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig18mDebugInputTensorsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::mDebugInputTensors</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig19mDebugOutputTensorsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::mDebugOutputTensors</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig17mDebugTensorNamesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::mDebugTensorNames</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig26mDebugTensorsMaxIterationsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugConfig::mDebugTensorsMaxIterations</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEENSt8optionalI11EagleConfigEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::DecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::setDecodingMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::getDecodingMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::setLookaheadDecoding()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::getLookaheadDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::setMedusaChoices()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::getMedusaChoices()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14setEagleConfigERK11EagleConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::setEagleConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig14getEagleConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::getEagleConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::mDecodingMode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::mLookaheadDecodingConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::mMedusaChoices</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig12mEagleConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingConfig::mEagleConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig18DynamicBatchConfigEbb10SizeType32NSt6vectorINSt4pairI10SizeType3210SizeType32EEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::DynamicBatchConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig34getDynamicBatchMovingAverageWindowEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::getDynamicBatchMovingAverageWindow()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig24getEnableBatchSizeTuningEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::getEnableBatchSizeTuning()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig27getEnableMaxNumTokensTuningEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::getEnableMaxNumTokensTuning()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig17getBatchSizeTableEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::getBatchSizeTable()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig39kDefaultDynamicBatchMovingAverageWindowE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::kDefaultDynamicBatchMovingAverageWindow</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22kDefaultBatchSizeTableE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::kDefaultBatchSizeTable</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22mEnableBatchSizeTuningE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::mEnableBatchSizeTuning</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig25mEnableMaxNumTokensTuningE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::mEnableMaxNumTokensTuning</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig32mDynamicBatchMovingAverageWindowE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::mDynamicBatchMovingAverageWindow</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig15mBatchSizeTableE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DynamicBatchConfig::mBatchSizeTable</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig11EagleConfigENSt8optionalI12EagleChoicesEEbNSt8optionalIfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::EagleConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfigeqERK11EagleConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfig15getEagleChoicesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::getEagleChoices()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfig21getPosteriorThresholdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::getPosteriorThreshold()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfig16isGreedySamplingEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::isGreedySampling()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig19checkPosteriorValueERKNSt8optionalIfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::checkPosteriorValue()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig13mEagleChoicesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::mEagleChoices</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig15mGreedySamplingE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::mGreedySampling</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig19mPosteriorThresholdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleConfig::mPosteriorThreshold</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfigRKNSt8optionalINSt3mapINSt6stringE6TensorEEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutorD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::~Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK8Executor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutoraSERK8Executor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERR8Executor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::Executor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutoraSERR8Executor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::enqueueRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::enqueueRequests()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::awaitResponses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::awaitResponses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::awaitResponses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::getNumResponsesReady()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor13cancelRequestE6IdType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::cancelRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8shutdownEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::shutdown()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::getLatestIterationStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::getLatestRequestStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor21getLatestDebugTensorsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::getLatestDebugTensors()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::canEnqueueRequests()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Executor13isParticipantEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::isParticipant()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Executor22getKVCacheEventManagerEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::getKVCacheEventManager()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor5mImplE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Executor::mImpl</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType3215SchedulerConfig13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI25LogitsPostProcessorConfigEENSt8optionalI14DecodingConfigEEfNSt8optionalI10SizeType32EERK29ExtendedRuntimePerfKnobConfigNSt8optionalI11DebugConfigEE10SizeType328uint64_tNSt8optionalI25SpeculativeDecodingConfigEENSt8optionalI20GuidedDecodingConfigEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::ExecutorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getMaxBeamWidth()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getSchedulerConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getKvCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21getSchedulerConfigRefEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getSchedulerConfigRef()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig19getKvCacheConfigRefEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getKvCacheConfigRef()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getEnableChunkedContext()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getNormalizeLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getIterStatsMaxIterations()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getRequestStatsMaxIterations()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getBatchingType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getMaxBatchSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getMaxNumTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getParallelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getPeftCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getLogitsPostProcessorConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getLogitsPostProcessorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getGpuWeightsPercent()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxQueueSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getMaxQueueSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig32getExtendedRuntimePerfKnobConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getExtendedRuntimePerfKnobConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig14getDebugConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getDebugConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig19getRecvPollPeriodMsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getRecvPollPeriodMs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getMaxSeqIdleMicrosecondsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getMaxSeqIdleMicroseconds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getSpecDecConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getSpecDecConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getGuidedDecodingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::getGuidedDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setMaxBeamWidth()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setMaxBatchSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setMaxNumTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setSchedulerConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setKvCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setEnableChunkedContext()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setNormalizeLogProbs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setIterStatsMaxIterations()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setRequestStatsMaxIterations()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setBatchingType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setParallelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setPeftCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setLogitsPostProcessorConfigERK25LogitsPostProcessorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setLogitsPostProcessorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setGpuWeightsPercent()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxQueueSizeERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setMaxQueueSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig32setExtendedRuntimePerfKnobConfigERK29ExtendedRuntimePerfKnobConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setExtendedRuntimePerfKnobConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14setDebugConfigERK11DebugConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setDebugConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig19setRecvPollPeriodMsERK10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setRecvPollPeriodMs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setMaxSeqIdleMicrosecondsE8uint64_t"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setMaxSeqIdleMicroseconds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setSpecDecConfigERK25SpeculativeDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setSpecDecConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setGuidedDecodingConfigERK20GuidedDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::setGuidedDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultMaxSeqIdleMicrosecondsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::kDefaultMaxSeqIdleMicroseconds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultIterStatsMaxIterationsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::kDefaultIterStatsMaxIterations</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig33kDefaultRequestStatsMaxIterationsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::kDefaultRequestStatsMaxIterations</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mMaxBeamWidth</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mSchedulerConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mKvCacheConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mEnableChunkedContext</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mNormalizeLogProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mIterStatsMaxIterations</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mRequestStatsMaxIterations</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mBatchingType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mMaxBatchSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mMaxNumTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mParallelConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mPeftCacheConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mLogitsPostProcessorConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mLogitsPostProcessorConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mDecodingConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mGpuWeightsPercent</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxQueueSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mMaxQueueSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30mExtendedRuntimePerfKnobConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mExtendedRuntimePerfKnobConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig12mDebugConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mDebugConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17mRecvPollPeriodMsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mRecvPollPeriodMs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mMaxSeqIdleMicrosecondsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mMaxSeqIdleMicroseconds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mSpeculativeDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mSpeculativeDecodingConfig</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mGuidedDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExecutorConfig::mGuidedDecodingConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig29ExtendedRuntimePerfKnobConfigEbbb10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::ExtendedRuntimePerfKnobConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigeqERK29ExtendedRuntimePerfKnobConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17getMultiBlockModeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getMultiBlockMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27getEnableContextFMHAFP32AccEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getEnableContextFMHAFP32Acc()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16getCudaGraphModeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getCudaGraphMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21getCudaGraphCacheSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getCudaGraphCacheSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17setMultiBlockModeEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setMultiBlockMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27setEnableContextFMHAFP32AccEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setEnableContextFMHAFP32Acc()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16setCudaGraphModeEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setCudaGraphMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21setCudaGraphCacheSizeE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setCudaGraphCacheSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig15mMultiBlockModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mMultiBlockMode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig25mEnableContextFMHAFP32AccE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mEnableContextFMHAFP32Acc</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig14mCudaGraphModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mCudaGraphMode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig19mCudaGraphCacheSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mCudaGraphCacheSize</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEERKNSt8optionalIbEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::ExternalDraftTokensConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::getTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::getLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::getAcceptanceThreshold()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig13getFastLogitsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::getFastLogits()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::mTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::mLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::mAcceptanceThreshold</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig11mFastLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ExternalDraftTokensConfig::mFastLogits</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingBackend</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig20GuidedDecodingConfigE21GuidedDecodingBackendNSt8optionalINSt6vectorINSt6stringEEEEENSt8optionalINSt6stringEEENSt8optionalINSt6vectorI11TokenIdTypeEEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfigeqERK20GuidedDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig10setBackendERK21GuidedDecodingBackend"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::setBackend()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig10getBackendEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::getBackend()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setEncodedVocabERKNSt6vectorINSt6stringEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::setEncodedVocab()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getEncodedVocabEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::getEncodedVocab()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setTokenizerStrERKNSt6stringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::setTokenizerStr()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getTokenizerStrEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::getTokenizerStr()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setStopTokenIdsERKNSt6vectorI11TokenIdTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::setStopTokenIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getStopTokenIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::getStopTokenIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig8validateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::validate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig8mBackendE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::mBackend</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mEncodedVocabE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::mEncodedVocab</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mTokenizerStrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::mTokenizerStr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mStopTokenIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingConfig::mStopTokenIds</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::GuideType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams20GuidedDecodingParamsE9GuideTypeNSt8optionalINSt6stringEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::GuidedDecodingParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParamseqERK20GuidedDecodingParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParams12getGuideTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::getGuideType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParams8getGuideEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::getGuide()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams10mGuideTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::mGuideType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams6mGuideE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::GuidedDecodingParams::mGuide</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17JsonSerializationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::JsonSerialization</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::JsonSerialization::toJsonStr()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::JsonSerialization::toJsonStr()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::JsonSerialization::toJsonStr()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEbRKNSt8optionalI9FloatTypeEENSt8optionalI17RetentionPriorityEE6size_tRKNSt8optionalIN12tensorrt_llm7runtime15RuntimeDefaultsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::KvCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getEnableBlockReuse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getMaxTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getMaxAttentionWindowVecEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getMaxAttentionWindowVec()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getSinkTokenLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getFreeGpuMemoryFraction()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig23getCrossKvCacheFractionEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getCrossKvCacheFraction()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getHostCacheSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getOnboardBlocks()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig30getSecondaryOffloadMinPriorityEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getSecondaryOffloadMinPriority()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig21getEventBufferMaxSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::getEventBufferMaxSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setEnableBlockReuse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setMaxTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setMaxAttentionWindowVecENSt6vectorI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setMaxAttentionWindowVec()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setSinkTokenLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setFreeGpuMemoryFraction()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig23setCrossKvCacheFractionE9FloatType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setCrossKvCacheFraction()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setHostCacheSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setOnboardBlocks()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig30setSecondaryOffloadMinPriorityENSt8optionalI17RetentionPriorityEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setSecondaryOffloadMinPriority()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig21setEventBufferMaxSizeE6size_t"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::setEventBufferMaxSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig34fillEmptyFieldsFromRuntimeDefaultsEN12tensorrt_llm7runtime15RuntimeDefaultsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::fillEmptyFieldsFromRuntimeDefaults()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mEnableBlockReuse</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mMaxTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mMaxAttentionWindowVecE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mMaxAttentionWindowVec</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mSinkTokenLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mFreeGpuMemoryFraction</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig21mCrossKvCacheFractionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mCrossKvCacheFraction</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mHostCacheSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mOnboardBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig28mSecondaryOffloadMinPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mSecondaryOffloadMinPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig19mEventBufferMaxSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheConfig::mEventBufferMaxSize</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheCreatedDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheCreatedData</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheCreatedData22numBlocksPerCacheLevelE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheCreatedData::numBlocksPerCacheLevel</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEventE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEvent</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEvent12KVCacheEventE6IdType16KVCacheEventData"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEvent::KVCacheEvent()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEvent7eventIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEvent::eventId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEvent4dataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEvent::data</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventDiff</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDiff8oldValueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventDiff::oldValue</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDiff8newValueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventDiff::newValue</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManagerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventManager</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManager19KVCacheEventManagerENSt10shared_ptrIN12tensorrt_llm13batch_manager16kv_cache_manager18BaseKVCacheManagerEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventManager::KVCacheEventManager()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManager15getLatestEventsENSt8optionalINSt6chrono12millisecondsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventManager::getLatestEvents()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManager14kvCacheManagerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheEventManager::kvCacheManager</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheRemovedDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheRemovedData</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheRemovedData11blockHashesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheRemovedData::blockHashes</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::KvCacheRetentionConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigERKNSt6vectorI25TokenRangeRetentionConfigEE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::KvCacheRetentionConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig29getTokenRangeRetentionConfigsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::getTokenRangeRetentionConfigs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig26getDecodeRetentionPriorityEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::getDecodeRetentionPriority()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig19getDecodeDurationMsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::getDecodeDurationMs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig36getPerBlockRetentionPriorityDurationE10SizeType3210SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::getPerBlockRetentionPriorityDuration()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig21kMinRetentionPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::kMinRetentionPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig21kMaxRetentionPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::kMaxRetentionPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::kDefaultRetentionPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig27mTokenRangeRetentionConfigsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::mTokenRangeRetentionConfigs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig24mDecodeRetentionPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::mDecodeRetentionPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig17mDecodeDurationMsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::mDecodeDurationMs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData22KVCacheStoredBlockDataE6IdTypeN12tensorrt_llm7runtime15VecUniqueTokensEN12tensorrt_llm7runtime14LoraTaskIdTypeE10SizeType3210SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData::KVCacheStoredBlockData()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData9blockHashE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData::blockHash</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData6tokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData::tokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData6loraIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData::loraId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData10cacheLevelE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData::cacheLevel</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData8priorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredBlockData::priority</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredData</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredData10parentHashE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredData::parentHash</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredData6blocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheStoredData::blocks</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedDataE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData18KVCacheUpdatedDataE6IdType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData::KVCacheUpdatedData()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData17cacheLevelUpdatedE10SizeType3210SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData::cacheLevelUpdated()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData15priorityUpdatedE10SizeType3210SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData::priorityUpdated()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData9blockHashE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData::blockHash</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData10cacheLevelE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData::cacheLevel</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData8priorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KVCacheUpdatedData::priority</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig25LogitsPostProcessorConfigENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::LogitsPostProcessorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig15getProcessorMapEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::getProcessorMap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig19getProcessorBatchedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::getProcessorBatched()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig12getReplicateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::getReplicate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig15setProcessorMapERK22LogitsPostProcessorMap"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::setProcessorMap()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig19setProcessorBatchedERK26LogitsPostProcessorBatched"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::setProcessorBatched()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig12setReplicateEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::setReplicate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig13mProcessorMapE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::mProcessorMap</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig17mProcessorBatchedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::mProcessorBatched</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig10mReplicateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorConfig::mReplicate</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::LookaheadDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::LookaheadDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::get()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::getWindowSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::getNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::getVerificationSetSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::calculateSpeculativeResource()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::isLE()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::isLegal()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::mWindowSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::mNgramSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LookaheadDecodingConfig::mVerificationSetSize</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::LoraConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::getTaskId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig10getWeightsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::getWeights()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig9getConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::getConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig7mTaskIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::mTaskId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig8mWeightsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::mWeights</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig7mConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LoraConfig::mConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MropeConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfig11MropeConfigE6Tensor10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MropeConfig::MropeConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11MropeConfig20getMRopeRotarySinCosEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MropeConfig::getMRopeRotarySinCos()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor11MropeConfig22getMRopePositionDeltasEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MropeConfig::getMRopePositionDeltas()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfig18mMRopeRotarySinCosE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MropeConfig::mMRopeRotarySinCos</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfig20mMRopePositionDeltasE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MropeConfig::mMRopePositionDeltas</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::OrchestratorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::getIsOrchestrator()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::getWorkerExecutablePath()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::getOrchLeaderComm()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getSpawnProcessesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::getSpawnProcesses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::setIsOrchestrator()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::setWorkerExecutablePath()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::setOrchLeaderComm()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setSpawnProcessesEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::setSpawnProcesses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::mIsOrchestrator</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::mWorkerExecutablePath</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::mOrchLeaderComm</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mSpawnProcessesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OrchestratorConfig::mSpawnProcesses</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbbb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::OutputConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig14returnLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::returnLogProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::returnContextLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::returnGenerationLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::excludeInputFromOutput</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::returnEncoderOutput</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig17returnPerfMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::OutputConfig::returnPerfMetrics</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::ParallelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::getCommunicationType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::getCommunicationMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::getDeviceIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::getParticipantIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::getOrchestratorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::setCommunicationType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::setCommunicationMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::setDeviceIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::setParticipantIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::setOrchestratorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::mCommType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::mCommMode</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::mDeviceIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::mParticipantIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ParallelConfig::mOrchestratorConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::PeftCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getNumHostModuleLayer()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getNumDeviceModuleLayer()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getOptimalAdapterSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getMaxAdapterSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getNumPutWorkers()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getNumEnsureWorkers()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getNumCopyStreams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getMaxPagesPerBlockHost()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getMaxPagesPerBlockDevice()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getDeviceCachePercent()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::getHostCacheSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig26kDefaultOptimalAdapterSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::kDefaultOptimalAdapterSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig22kDefaultMaxAdapterSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxAdapterSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig28kDefaultMaxPagesPerBlockHostE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxPagesPerBlockHost</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig30kDefaultMaxPagesPerBlockDeviceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxPagesPerBlockDevice</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mNumHostModuleLayer</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mNumDeviceModuleLayer</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mOptimalAdapterSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mMaxAdapterSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mNumPutWorkers</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mNumEnsureWorkers</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mNumCopyStreams</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mMaxPagesPerBlockHost</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mMaxPagesPerBlockDevice</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mDeviceCachePercent</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PeftCacheConfig::mHostCacheSize</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PromptTuningConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6TensorNSt8optionalI16VecTokenExtraIdsEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PromptTuningConfig::PromptTuningConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PromptTuningConfig::getEmbeddingTable()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig21getInputTokenExtraIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PromptTuningConfig::getInputTokenExtraIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PromptTuningConfig::mEmbeddingTable</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig19mInputTokenExtraIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PromptTuningConfig::mInputTokenExtraIds</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI11MropeConfigEENSt8optionalI10LoraConfigEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI22KvCacheRetentionConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEENSt8optionalI6IdTypeEEb12PriorityType11RequestTypeNSt8optionalI18ContextPhaseParamsEENSt8optionalI6TensorEENSt8optionalI10SizeType32EENSt8optionalI6TensorEE10SizeType32NSt8optionalI11EagleConfigEENSt8optionalI6TensorEENSt8optionalI20GuidedDecodingParamsEENSt8optionalI16MillisecondsTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::Request()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::Request()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::Request()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestaSERK7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestaSERR7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::~Request()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request16getInputTokenIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getInputTokenIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request12getMaxTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getMaxTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request15getMaxNewTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getMaxNewTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request12getStreamingEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getStreaming()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request17getSamplingConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getSamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request15getOutputConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getOutputConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request8getEndIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getEndId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request8getPadIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getPadId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request14getPositionIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getPositionIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request11getBadWordsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getBadWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request12getStopWordsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getStopWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getEmbeddingBias()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getExternalDraftTokensConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getPromptTuningConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request14getMropeConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getMropeConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request13getLoraConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getLoraConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request18getLookaheadConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getLookaheadConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request25getKvCacheRetentionConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getKvCacheRetentionConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getLogitsPostProcessorName()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getEncoderInputTokenIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request11getClientIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getClientId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request11getPriorityEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getPriority()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request27getReturnAllGeneratedTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getReturnAllGeneratedTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request21getContextPhaseParamsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputFeaturesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getEncoderInputFeatures()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request22getEncoderOutputLengthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getEncoderOutputLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request21getCrossAttentionMaskEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getCrossAttentionMask()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request14getRequestTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getRequestType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request21getNumReturnSequencesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getNumReturnSequences()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request14getEagleConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getEagleConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request22getSkipCrossAttnBlocksEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getSkipCrossAttnBlocks()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request23getGuidedDecodingParamsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getGuidedDecodingParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor7Request17getAllottedTimeMsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::getAllottedTimeMs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request12setStreamingEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setStreaming()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setSamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setOutputConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request8setEndIdE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setEndId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request8setPadIdE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setPadId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request14setPositionIdsERKNSt6vectorI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setPositionIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setBadWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setStopWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setEmbeddingBias()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setExternalDraftTokensConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setPromptTuningConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request14setMropeConfigERK11MropeConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setMropeConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setLoraConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request18setLookaheadConfigERK23LookaheadDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setLookaheadConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request25setKvCacheRetentionConfigERK22KvCacheRetentionConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setKvCacheRetentionConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setLogitsPostProcessorName()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setEncoderInputTokenIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request11setClientIdE6IdType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setClientId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request11setPriorityE12PriorityType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setPriority()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request27setReturnAllGeneratedTokensEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setReturnAllGeneratedTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request14setRequestTypeERK11RequestType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setRequestType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request21setContextPhaseParamsE18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputFeaturesE6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setEncoderInputFeatures()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request22setEncoderOutputLengthE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setEncoderOutputLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request21setCrossAttentionMaskE6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setCrossAttentionMask()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request21setNumReturnSequencesE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setNumReturnSequences()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request14setEagleConfigERKNSt8optionalI11EagleConfigEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setEagleConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request22setSkipCrossAttnBlocksE6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setSkipCrossAttnBlocks()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request23setGuidedDecodingParamsERK20GuidedDecodingParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setGuidedDecodingParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request17setAllottedTimeMsE16MillisecondsType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::setAllottedTimeMs()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request16kDefaultPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::kDefaultPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::kBatchedPostProcessorName</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request5mImplE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Request::mImpl</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringENSt8optionalI6IdTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::Response()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6ResultNSt8optionalI6IdTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::Response()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::~Response()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::Response()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::Response()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseaSERK8Response"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseaSERR8Response"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Response12getRequestIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::getRequestId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Response11getClientIdEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::getClientId()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Response8hasErrorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::hasError()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Response11getErrorMsgEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::getErrorMsg()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor8Response9getResultEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::getResult()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response5mImplE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Response::mImpl</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result7isFinalE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::isFinal</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result14outputTokenIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::outputTokenIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result11cumLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::cumLogProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result8logProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::logProbs</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result13contextLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::contextLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result16generationLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::generationLogits</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result21specDecFastLogitsInfoE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::specDecFastLogitsInfo</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result13encoderOutputE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::encoderOutput</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result13finishReasonsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::finishReasons</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result18contextPhaseParamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::contextPhaseParams</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result12decodingIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::decodingIter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result13sequenceIndexE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::sequenceIndex</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result15isSequenceFinalE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::isSequenceFinal</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Result18requestPerfMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Result::requestPerfMetrics</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDurationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RetentionPriorityAndDuration</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration28RetentionPriorityAndDurationERKNSt8optionalI17RetentionPriorityEERKNSt8optionalINSt6chrono12millisecondsEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RetentionPriorityAndDuration::RetentionPriorityAndDuration()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration17retentionPriorityE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RetentionPriorityAndDuration::retentionPriority</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration10durationMsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RetentionPriorityAndDuration::durationMs</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::SamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getBeamWidth()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig17getNumReturnBeamsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getNumReturnBeams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopKEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getTopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getTopPMin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getTopPResetIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getTopPDecay()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getSeedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getSeed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getRandomSeed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getMinTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getMinLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getBeamSearchDiversityRate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getPresencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getFrequencyPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getLengthPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getEarlyStopping()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig21getNumReturnSequencesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::getNumReturnSequences()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setBeamWidth()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setTopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setTopPMin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setTopPResetIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setTopPDecay()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setSeedERKNSt8optionalI14RandomSeedTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setSeed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setRandomSeed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinTokensERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setMinTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setMinLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setBeamSearchDiversityRate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setPresencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setFrequencyPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setLengthPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setEarlyStopping()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig21setNumReturnSequencesERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::setNumReturnSequences()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20updateNumReturnBeamsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::updateNumReturnBeams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mBeamWidth</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopKE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mTopK</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopPE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mTopP</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig8mTopPMinE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mTopPMin</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mTopPResetIds</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mTopPDecay</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mSeedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mSeed</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12mTemperatureE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mTemperature</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mMinTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mMinTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mBeamSearchDiversityRate</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mRepetitionPenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mPresencePenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mFrequencyPenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mLengthPenalty</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mEarlyStopping</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mNoRepeatNgramSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig19mNumReturnSequencesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mNumReturnSequences</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig15mNumReturnBeamsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::mNumReturnBeams</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkBeamWidth()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkTopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkTopPMin()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkTopPResetIds()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkTopPDecay()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkMinTokensERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkMinTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkBeamSearchDiversityRate()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig23checkNumReturnSequencesERKNSt8optionalI10SizeType32EE10SizeType32"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SamplingConfig::checkNumReturnSequences()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEENSt8optionalI18DynamicBatchConfigEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::SchedulerConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfigeqERK15SchedulerConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::getCapacitySchedulerPolicy()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::getContextChunkingPolicy()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig21getDynamicBatchConfigEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::getDynamicBatchConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::mCapacitySchedulerPolicy</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::mContextChunkingPolicy</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig19mDynamicBatchConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SchedulerConfig::mDynamicBatchConfig</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingConfig</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfig25SpeculativeDecodingConfigEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingConfig::SpeculativeDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor25SpeculativeDecodingConfigeqERK25SpeculativeDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingConfig::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfig10fastLogitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingConfig::fastLogits</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo8toTensorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo::toTensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo14draftRequestIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo::draftRequestId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo18draftParticipantIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo::draftParticipantId</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::mpi</span></code></a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#serialization-h">serialization.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13SerializationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeTimePointERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeTimePoint()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN18RequestPerfMetrics9TimePointERNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN18RequestPerfMetrics9TimePointE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeRequestPerfMetricsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequestPerfMetrics()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18RequestPerfMetricsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18RequestPerfMetrics"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeOutputConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeExternalDraftTokensConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializePromptTuningConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeMropeConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeMropeConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11MropeConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11MropeConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeLoraConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeCommStateERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeCommState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache9CommStateERNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache9CommStateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeSocketStateERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSocketState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache11SocketStateERNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache11SocketStateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization21deserializeCacheStateERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeCacheState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache10CacheStateERNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache10CacheStateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization31deserializeDataTransceiverStateERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDataTransceiverState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20DataTransceiverStateRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20DataTransceiverState"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeContextPhaseParamsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeContextPhaseParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18ContextPhaseParamsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18ContextPhaseParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeTensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization32deserializeSpecDecFastLogitsInfoERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSpecDecFastLogitsInfo()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK33SpeculativeDecodingFastLogitsInfoRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK33SpeculativeDecodingFastLogitsInfo"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeResult()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeResponse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeResponses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeKvCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeDynamicBatchConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDynamicBatchConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18DynamicBatchConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18DynamicBatchConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSchedulerConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization40deserializeExtendedRuntimePerfKnobConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeExtendedRuntimePerfKnobConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK29ExtendedRuntimePerfKnobConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK29ExtendedRuntimePerfKnobConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeParallelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializePeftCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeOrchestratorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDecodingMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeLookaheadDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeEagleConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeEagleConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11EagleConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11EagleConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeSpeculativeDecodingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSpeculativeDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25SpeculativeDecodingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25SpeculativeDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeGuidedDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingParamsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeGuidedDecodingParams()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingParamsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingParams"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization33deserializeKvCacheRetentionConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeKvCacheRetentionConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK22KvCacheRetentionConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK22KvCacheRetentionConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeTokenRangeRetentionConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeTokenRangeRetentionConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigERNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeDebugConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDebugConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11DebugConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11DebugConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeExecutorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeKvCacheStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeStaticBatchingStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeInflightBatchingStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeIterationStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeIterationStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI14IterationStatsEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization28deserializeIterationStatsVecERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeIterationStatsVec()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization33deserializeDisServingRequestStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDisServingRequestStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK22DisServingRequestStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK22DisServingRequestStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeRequestStageERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequestStage()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12RequestStageRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStage"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeRequestStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequestStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12RequestStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequestStatsPerIteration()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequestStatsPerIteration()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIterationRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIteration"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK24RequestStatsPerIteration"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI24RequestStatsPerIterationEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization38deserializeRequestStatsPerIterationVecERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequestStatsPerIterationVec()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeString()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeBool()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeModelType()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::kv_cache</span></code></a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#tensor-h">tensor.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Base</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::DimType64</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Shape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Shape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Shape()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::CudaStreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToCpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToPooledPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToManaged()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToGpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::~Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERR6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7getDataEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getData()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getDataEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getData()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor11getDataTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getDataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getMemoryType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor8getShapeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getShape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getSizeInBytes()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::setZero()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::setFrom()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6TensorcvbEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator</span> <span class="pre">bool()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6TensoreqERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6TensorneERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator!=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::cpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3cpuE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::cpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor6pinnedE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pooledPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pooledPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::managed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor7managedE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::managed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::gpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3gpuE6Tensor13CudaStreamPtr5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::gpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::of()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::of()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::of()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Impl</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyTo()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7mTensorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::mTensor</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeE8DataTypev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getRuntimeType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::toITensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::ofITensor()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detailE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail::DimType64</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9toITensorERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail::toITensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail::ofITensor()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime</span></code></a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#types-h">types.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TensorPtr</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SizeType32</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FloatType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TokenIdType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::VecTokens</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BeamTokens</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IdType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16VecTokenExtraIdsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::VecTokenExtraIds</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RandomSeedType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::VecLogProbs</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StreamPtr</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16MillisecondsTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MillisecondsType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessor</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorMap</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorBatched</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MedusaChoices</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12EagleChoicesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::EagleChoices</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12PriorityTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PriorityType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BufferViewE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BufferView</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kBOOL</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kUINT8</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kINT8</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kINT32</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kINT64</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBF16E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kBF16</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType4kFP8E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kFP8</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP16E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kFP16</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP32E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kFP32</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType8kUNKNOWNE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kUNKNOWN</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestType35REQUEST_TYPE_CONTEXT_AND_GENERATIONE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestType::REQUEST_TYPE_CONTEXT_AND_GENERATION</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestType25REQUEST_TYPE_CONTEXT_ONLYE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestType::REQUEST_TYPE_CONTEXT_ONLY</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestType28REQUEST_TYPE_GENERATION_ONLYE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestType::REQUEST_TYPE_GENERATION_ONLY</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kCPUE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kCPU</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kCPU_PINNED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType15kCPU_PINNEDPOOLE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kCPU_PINNEDPOOL</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kGPUE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kGPU</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kUVME"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kUVM</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType8kUNKNOWNE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kUNKNOWN</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType::kDECODER_ONLY</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType::kENCODER_ONLY</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType::kENCODER_DECODER</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BatchingType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingType7kSTATICE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BatchingType::kSTATIC</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BatchingType::kINFLIGHT</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy::kMAX_UTILIZATION</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy13kSTATIC_BATCHE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy::kSTATIC_BATCH</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextChunkingPolicy</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextChunkingPolicy::kFIRST_COME_FIRST_SERVED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextChunkingPolicy::kEQUAL_PROGRESS</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationType::kMPI</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationMode</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationMode::kLEADER</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationMode::kORCHESTRATOR</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage7kQUEUEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kQUEUED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kENCODER_IN_PROGRESS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kCONTEXT_IN_PROGRESS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kGENERATION_IN_PROGRESS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kGENERATION_COMPLETE</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReasonE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReason13kNOT_FINISHEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason::kNOT_FINISHED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReason7kEND_IDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason::kEND_ID</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReason11kSTOP_WORDSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason::kSTOP_WORDS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReason7kLENGTHE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason::kLENGTH</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReason10kTIMED_OUTE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason::kTIMED_OUT</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReason10kCANCELLEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FinishReason::kCANCELLED</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIterationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugTensorsPerIteration</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIteration4iterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugTensorsPerIteration::iter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIteration12debugTensorsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DebugTensorsPerIteration::debugTensors</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::UnderlyingType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14useTemperatureEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useOccurrencePenalties()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::usePresencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useFrequencyPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useMinLengthEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useMinLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useBanTokensEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useBanTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11useBanWordsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useBanWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useStopWordsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useStopWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useMaxLengthStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useExplicitEosStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isAutoEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isAuto()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopKEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopKorTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopKandTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isBeamSearch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8isMedusaEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isMedusa()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isLookahead()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isExplicitDraftTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExternalDraftTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isExternalDraftTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode7isEagleEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isEagle()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUsePresencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseFrequencyPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseMinLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseOccurrencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUsePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseBanWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseBanTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseStopWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseMaxLengthStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseExplicitEosStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseStopCriteria()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::DecodingMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8getStateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::getState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4AutoEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Auto()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopKEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::TopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::TopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8TopKTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::TopKTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10BeamSearchEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::BeamSearch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6MedusaEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Medusa()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9LookaheadEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Lookahead()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::ExplicitDraftTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19ExternalDraftTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::ExternalDraftTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5EagleEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Eagle()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::anyBitSet()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::allBitSet()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::setBitTo()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6mStateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::mState</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseRepetitionPenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseFrequencyPenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUsePresencePenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseTemperature</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseMinLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseBanWords</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseStopWords</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseMaxLengthStop</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseExplicitEosStop</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseNoRepeatNgramSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kStandardStopCriteria</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseOccurrencePenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUsePenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseBanTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kNumFlags</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kAutoE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kAuto</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kTopK</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kTopP</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11kBeamSearchE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kBeamSearch</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode7kMedusaE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kMedusa</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10kLookaheadE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kLookahead</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kExplicitDraftTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20kExternalDraftTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kExternalDraftTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6kEagleE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kEagle</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kTopKTopPE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kTopKTopP</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DisServingRequestStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStats17kvCacheTransferMSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DisServingRequestStats::kvCacheTransferMS</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numScheduledRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numContextRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numGenRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numPausedRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numCtxTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::microBatchId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::avgNumDecodedTokensPerIter</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats9timestampE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::timestamp</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats4iterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::iter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats13iterLatencyMSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::iterLatencyMS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats31newActiveRequestsQueueLatencyMSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::newActiveRequestsQueueLatencyMS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20numNewActiveRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::numNewActiveRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numActiveRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::numActiveRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::numQueuedRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20numCompletedRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::numCompletedRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxNumActiveRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats18maxBatchSizeStaticE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxBatchSizeStatic</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats28maxBatchSizeTunerRecommendedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxBatchSizeTunerRecommended</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19maxBatchSizeRuntimeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxBatchSizeRuntime</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats18maxNumTokensStaticE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxNumTokensStatic</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats28maxNumTokensTunerRecommendedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxNumTokensTunerRecommended</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19maxNumTokensRuntimeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxNumTokensRuntime</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11gpuMemUsageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::gpuMemUsage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11cpuMemUsageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::cpuMemUsage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::pinnedMemUsage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats12kvCacheStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::kvCacheStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::crossKvCacheStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::staticBatchingStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::inflightBatchingStats</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::maxNumBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::freeNumBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::usedNumBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::tokensPerBlock</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::allocTotalBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::allocNewBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::reusedBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12missedBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::missedBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12cacheHitRateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::cacheHitRate</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::TimePoint</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13timingMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::timingMetrics</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14kvCacheMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::kvCacheMetrics</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9firstIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::firstIter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics8lastIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::lastIter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics4iterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::iter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetricsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats2idE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::id</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats5stageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::stage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::contextPrefillPosition</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::numGeneratedTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::avgNumDecodedTokensPerIter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats9scheduledE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::scheduled</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats6pausedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::paused</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats15disServingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::disServingStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats26allocTotalBlocksPerRequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::allocTotalBlocksPerRequest</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats24allocNewBlocksPerRequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::allocNewBlocksPerRequest</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22reusedBlocksPerRequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::reusedBlocksPerRequest</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22missedBlocksPerRequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::missedBlocksPerRequest</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats24kvCacheHitRatePerRequestE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::kvCacheHitRatePerRequest</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStatsPerIteration</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration4iterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStatsPerIteration::iter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStatsPerIteration::requestStats</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numScheduledRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numContextRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numCtxTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numGenTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::emptyGenSlots</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0_bEN12tensorrt_llm8executor10TypeTraitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIbEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIbE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIfE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsI4halfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19PhonyNameDueToError5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PhonyNameDueToError::value</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#version-h">version.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19kTensorRtLlmVersionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::kTensorRtLlmVersion</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="runtime.html">Runtime</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Command-Line Reference</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../commands/trtllm-build.html">trtllm-build</a></li>
<li class="toctree-l1"><a class="reference internal" href="../commands/trtllm-serve.html">trtllm-serve</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Architecture</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../architecture/overview.html">TensorRT-LLM Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html">Model Definition</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html#compilation">Compilation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html#runtime">Runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html#multi-gpu-and-multi-node-support">Multi-GPU and Multi-Node Support</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/checkpoint.html">TensorRT-LLM Checkpoint</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/workflow.html">TensorRT-LLM Build Workflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/add-model.html">Adding a Model</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Advanced</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../advanced/gpt-attention.html">Multi-Head, Multi-Query, and Group-Query Attention</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/gpt-runtime.html">C++ GPT Runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/executor.html">Executor API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/graph-rewriting.html">Graph Rewriting Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/inference-request.html">Inference Request</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/inference-request.html#responses">Responses</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/lora.html">Run gpt-2b + LoRA using GptManager / cpp runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/expert-parallelism.html">Expert Parallelism in TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/kv-cache-reuse.html">KV cache reuse</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/speculative-decoding.html">Speculative Sampling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/disaggregated-service.html">Disaggregated-Service (experimental)</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Performance</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-benchmarking.html">Benchmarking</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-best-practices.html">Best Practices</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-analysis.html">Performance Analysis</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../reference/troubleshooting.html">Troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/support-matrix.html">Support Matrix</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/precision.html">Numerical Precision</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/memory.html">Memory Usage of TensorRT-LLM</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Blogs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../blogs/H100vsA100.html">H100 has 4.6x A100 Performance in TensorRT-LLM, achieving 10,000 tok/s at 100ms to first token</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/H200launch.html">H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/Falcon180B-H200.html">Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/quantization-in-TRT-LLM.html">Speed up inference with SOTA quantization techniques in TRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/XQA-kernel.html">New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">tensorrt_llm</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active">Executor</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/_cpp_gen/executor.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="executor">
<h1>Executor<a class="headerlink" href="#executor" title="Link to this heading"></a></h1>
<section id="disaggserverutil-h">
<h2>disaggServerUtil.h<a class="headerlink" href="#disaggserverutil-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv412tensorrt_llm">
<span id="_CPPv312tensorrt_llm"></span><span id="_CPPv212tensorrt_llm"></span><span id="tensorrt_llm"></span><span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><a class="headerlink" href="#_CPPv412tensorrt_llm" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executorE">
<span id="_CPPv3N12tensorrt_llm8executorE"></span><span id="_CPPv2N12tensorrt_llm8executorE"></span><span id="tensorrt_llm::executor"></span><span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executorE" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executorE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executorE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executorE"></span><span id="tensorrt_llm::executor::disagg_executor"></span><span class="target" id="namespacetensorrt__llm_1_1executor_1_1disagg__executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">disagg_executor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executorE" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorE"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DisaggExecutorOrchestrator</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator26DisaggExecutorOrchestratorERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorIN8executor14ExecutorConfigEEERKNSt6vectorIN8executor14ExecutorConfigEEEbb">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator26DisaggExecutorOrchestratorERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorIN8executor14ExecutorConfigEEERKNSt6vectorIN8executor14ExecutorConfigEEEbb"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator26DisaggExecutorOrchestratorERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorIN8executor14ExecutorConfigEEERKNSt6vectorIN8executor14ExecutorConfigEEEbb"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::DisaggExecutorOrchestrator__std::vector:std::filesystem::path:CR.std::vector:std::filesystem::path:CR.std::vector:executor::ExecutorConfig:CR.std::vector:executor::ExecutorConfig:CR.b.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1a805b4f7a36690f40856f115ff5fa3d86"></span><span class="sig-name descname"><span class="n"><span class="pre">DisaggExecutorOrchestrator</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">ctxEnginePaths</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">genEnginePaths</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE" title="tensorrt_llm::executor"><span class="n"><span class="pre">executor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">ctxExecutorConfigs</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE" title="tensorrt_llm::executor"><span class="n"><span class="pre">executor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">genExecutorConfigs</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">hasContextAwaitThreads</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">hasGenAwaitThreads</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator26DisaggExecutorOrchestratorERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorINSt10filesystem4pathEEERKNSt6vectorIN8executor14ExecutorConfigEEERKNSt6vectorIN8executor14ExecutorConfigEEEbb" title="Link to this definition"></a><br /></dt>
<dd><p>Constructs a <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator"><span class="std std-ref">DisaggExecutorOrchestrator</span></a> object. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>ctxEnginePaths</strong> – A vector of file paths to context engine files. </p></li>
<li><p><strong>genEnginePaths</strong> – A vector of file paths to generation engine files. </p></li>
<li><p><strong>ctxExecutorConfigs</strong> – A vector of <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1ExecutorConfig"><span class="std std-ref">ExecutorConfig</span></a> for context executors. </p></li>
<li><p><strong>genExecutorConfigs</strong> – A vector of <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1ExecutorConfig"><span class="std std-ref">ExecutorConfig</span></a> for generation executors. </p></li>
<li><p><strong>hasContextAwaitThreads</strong> – Whether or not there are threads that receive response for each generation executor. </p></li>
<li><p><strong>hasGenAwaitThreads</strong> – Whether or not there are threads that receive response for each generation executor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator14enqueueContextERKNSt6vectorIN5texec7RequestEEENSt8optionalIiEEb">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator14enqueueContextERKNSt6vectorIN5texec7RequestEEENSt8optionalIiEEb"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator14enqueueContextERKNSt6vectorIN5texec7RequestEEENSt8optionalIiEEb"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::enqueueContext__std::vector:texec::Request:CR.std::optional:i:.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1a070294bc1a93c30ef3545760a96610e1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">enqueueContext</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">texec</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Request</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requests</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">selectContextId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batch</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator14enqueueContextERKNSt6vectorIN5texec7RequestEEENSt8optionalIiEEb" title="Link to this definition"></a><br /></dt>
<dd><p>Enqueue context-only requests to context executors. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>requests</strong> – A vector of context-only requests. </p></li>
<li><p><strong>selectContextId</strong> – The index of the context executor to use. If <code class="docutils literal notranslate"><span class="pre">std::nullopt</span></code>, the executor that has the smallest number of inflight requests will be used. </p></li>
<li><p><strong>batch</strong> – If true,enqueue requests in same context executor.If false, will try to use a different executor for each request. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of global request ids, corresponding to the order of the requests in <code class="docutils literal notranslate"><span class="pre">requests</span></code>, the id returned may be different from the request id in each executor. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator17enqueueGenerationERKNSt6vectorIN5texec7RequestEEERKNSt6vectorI6IdTypeEENSt8optionalIiEEb">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator17enqueueGenerationERKNSt6vectorIN5texec7RequestEEERKNSt6vectorI6IdTypeEENSt8optionalIiEEb"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator17enqueueGenerationERKNSt6vectorIN5texec7RequestEEERKNSt6vectorI6IdTypeEENSt8optionalIiEEb"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::enqueueGeneration__std::vector:texec::Request:CR.std::vector:IdType:CR.std::optional:i:.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1ae21ddfaf38813eb8a9d50ee0a6d81344"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">enqueueGeneration</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">texec</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Request</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requests</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">globalRequestIds</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">selectGenIdx</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batch</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator17enqueueGenerationERKNSt6vectorIN5texec7RequestEEERKNSt6vectorI6IdTypeEENSt8optionalIiEEb" title="Link to this definition"></a><br /></dt>
<dd><p>Enqueue generation-only requests to generation executors. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>requests</strong> – A vector of generation-only requests. </p></li>
<li><p><strong>globalRequestIds</strong> – A vector of global request ids, corresponding to the order of the requests,and must be the ids returned by the enqueueContext function. </p></li>
<li><p><strong>selectGenIdx</strong> – The index of the generation executor to use. If <code class="docutils literal notranslate"><span class="pre">std::nullopt</span></code>, the executor that has the smallest number of inflight requests will be used. </p></li>
<li><p><strong>batch</strong> – If true,enqueue requests in same generation executor.If false, will try to use a different executor for each request. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator21awaitContextResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator21awaitContextResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator21awaitContextResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::awaitContextResponses__std::optional:std::chrono::milliseconds:CR.std::optional:i:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1add36b49f7e4ba267bf310413cb56b455"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="tensorrt_llm::executor::disagg_executor::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitContextResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">contextIdx</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator21awaitContextResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for context responses. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>timeout</strong> – The maximum time to wait for new responses </p></li>
<li><p><strong>contextIdx</strong> – The index of the context executor to use. If <code class="docutils literal notranslate"><span class="pre">std::nullopt</span></code>, return ready responses in all context executors,if <code class="docutils literal notranslate"><span class="pre">hasContextAwaitThreads</span></code> is true, then this parameter must be std::nullopt. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of responses with corresponding global request ids </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator24awaitGenerationResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator24awaitGenerationResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator24awaitGenerationResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::awaitGenerationResponses__std::optional:std::chrono::milliseconds:CR.std::optional:i:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1a19bae735cb389e30f8baf0141a547b0a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="tensorrt_llm::executor::disagg_executor::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitGenerationResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">int</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">genIdx</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator24awaitGenerationResponsesERKNSt8optionalINSt6chrono12millisecondsEEENSt8optionalIiEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for generation responses. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>timeout</strong> – The maximum time to wait for new responses. </p></li>
<li><p><strong>genIdx</strong> – The index of the generation executor to use. If <code class="docutils literal notranslate"><span class="pre">std::nullopt</span></code>, return ready responses in all generation executors,if <code class="docutils literal notranslate"><span class="pre">hasGenAwaitThreads</span></code> is true, then this parameter must be std::nullopt. </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of responses with corresponding global request ids. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator10canEnqueueEv">
<span id="_CPPv3NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator10canEnqueueEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator10canEnqueueEv"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::canEnqueueC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1a63fdb48970256462e3180d018097ad92"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">canEnqueue</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator10canEnqueueEv" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if the current process is allowed to enqueueRequests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator19getContextExecutorsEv">
<span id="_CPPv3NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator19getContextExecutorsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator19getContextExecutorsEv"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::getContextExecutorsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1a6385de81f6584a23cfe8f1584ab206db"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">texec</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Executor</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getContextExecutors</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator19getContextExecutorsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get context executors. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator15getGenExecutorsEv">
<span id="_CPPv3NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator15getGenExecutorsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator15getGenExecutorsEv"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::getGenExecutorsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1a6d203b7d545eb0a5b7d0ff9f972205bd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">texec</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">Executor</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getGenExecutors</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator15getGenExecutorsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get generation executors. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorD0Ev"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::~DisaggExecutorOrchestrator"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1ae0b67aacedf99304a579568cdc3141f6"></span><span class="sig-name descname"><span class="n"><span class="pre">~DisaggExecutorOrchestrator</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestratorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator5mImplE"></span><span id="tensorrt_llm::executor::disagg_executor::DisaggExecutorOrchestrator::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1disagg__executor_1_1DisaggExecutorOrchestrator_1ac7f0969ddf0b6fef4c495414095aa6aa"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor26DisaggExecutorOrchestrator5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ResponseWithId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERRN12tensorrt_llm8executor8ResponseE6IdType">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERRN12tensorrt_llm8executor8ResponseE6IdType"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERRN12tensorrt_llm8executor8ResponseE6IdType"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::ResponseWithId__tensorrt_llm::executor::ResponseRR.IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a24568b6374b964b011a37252a872b480"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ResponseWithId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE" title="tensorrt_llm::executor"><span class="n"><span class="pre">executor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">response</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gid</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERRN12tensorrt_llm8executor8ResponseE6IdType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERKN12tensorrt_llm8executor8ResponseE6IdType">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERKN12tensorrt_llm8executor8ResponseE6IdType"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERKN12tensorrt_llm8executor8ResponseE6IdType"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::ResponseWithId__tensorrt_llm::executor::ResponseCR.IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a9b9497c2aa41a5d790d8ff7f385f10f1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ResponseWithId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE" title="tensorrt_llm::executor"><span class="n"><span class="pre">executor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">response</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">gid</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERKN12tensorrt_llm8executor8ResponseE6IdType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERR14ResponseWithId">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERR14ResponseWithId"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERR14ResponseWithId"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::ResponseWithId__ResponseWithIdRR"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a67bfcc9b54f71b7159c39e217f7dece8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ResponseWithId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERR14ResponseWithId" title="tensorrt_llm::executor::disagg_executor::ResponseWithId::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERR14ResponseWithId" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERK14ResponseWithId">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERK14ResponseWithId"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERK14ResponseWithId"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::ResponseWithId__ResponseWithIdCR"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a0b15ba9e365a1bfd3f2a2a5c0c8d18dd"></span><span class="sig-name descname"><span class="n"><span class="pre">ResponseWithId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERK14ResponseWithId" title="tensorrt_llm::executor::disagg_executor::ResponseWithId::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId14ResponseWithIdERK14ResponseWithId" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERR14ResponseWithId">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERR14ResponseWithId"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERR14ResponseWithId"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::assign-operator__ResponseWithIdRR"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a69d5eab7c0ed997cc298c0a39f6844ec"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="tensorrt_llm::executor::disagg_executor::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="tensorrt_llm::executor::disagg_executor::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERR14ResponseWithId" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERK14ResponseWithId">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERK14ResponseWithId"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERK14ResponseWithId"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::assign-operator__ResponseWithIdCR"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a187091dc435f65eaaf5a062429c1ce80"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="tensorrt_llm::executor::disagg_executor::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdE" title="tensorrt_llm::executor::disagg_executor::ResponseWithId"><span class="n"><span class="pre">ResponseWithId</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdaSERK14ResponseWithId" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithIdD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithIdD0Ev"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::~ResponseWithId"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a5042e819fe68dcf22e34869748154e3a"></span><span class="sig-name descname"><span class="n"><span class="pre">~ResponseWithId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithIdD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId8responseE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithId8responseE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithId8responseE"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::response__tensorrt_llm::executor::Response"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a61c708fecf2bf73f3e98b6b3c8b7e558"></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE" title="tensorrt_llm::executor"><span class="n"><span class="pre">executor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">response</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId8responseE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId3gidE">
<span id="_CPPv3N12tensorrt_llm8executor15disagg_executor14ResponseWithId3gidE"></span><span id="_CPPv2N12tensorrt_llm8executor15disagg_executor14ResponseWithId3gidE"></span><span id="tensorrt_llm::executor::disagg_executor::ResponseWithId::gid__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1disagg__executor_1_1ResponseWithId_1a39d756b6d5a76709a6fb505561a33c78"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gid</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15disagg_executor14ResponseWithId3gidE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="executor-h">
<h2>executor.h<a class="headerlink" href="#executor-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm13batch_managerE">
<span id="_CPPv3N12tensorrt_llm13batch_managerE"></span><span id="_CPPv2N12tensorrt_llm13batch_managerE"></span><span id="tensorrt_llm::batch_manager"></span><span class="target" id="namespacetensorrt__llm_1_1batch__manager"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">batch_manager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE">
<span id="_CPPv3N12tensorrt_llm13batch_manager16kv_cache_managerE"></span><span id="_CPPv2N12tensorrt_llm13batch_manager16kv_cache_managerE"></span><span id="tensorrt_llm::batch_manager::kv_cache_manager"></span><span class="target" id="namespacetensorrt__llm_1_1batch__manager_1_1kv__cache__manager"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kv_cache_manager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17RetentionPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor17RetentionPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor17RetentionPriorityE"></span><span class="target" id="executor_8h_1a7d47a118ea2835238c34ba65f7ac692e"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RetentionPriority</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor16KVCacheEventDataE">
<span id="_CPPv3N12tensorrt_llm8executor16KVCacheEventDataE"></span><span id="_CPPv2N12tensorrt_llm8executor16KVCacheEventDataE"></span><span class="target" id="executor_8h_1a2f045efaee75f7c0e1463d79434ad0b8"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheEventData</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">variant</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheCreatedDataE" title="tensorrt_llm::executor::KVCacheCreatedData"><span class="n"><span class="pre">KVCacheCreatedData</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredDataE" title="tensorrt_llm::executor::KVCacheStoredData"><span class="n"><span class="pre">KVCacheStoredData</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheRemovedDataE" title="tensorrt_llm::executor::KVCacheRemovedData"><span class="n"><span class="pre">KVCacheRemovedData</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedDataE" title="tensorrt_llm::executor::KVCacheUpdatedData"><span class="n"><span class="pre">KVCacheUpdatedData</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDataE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7versionEv">
<span id="_CPPv3N12tensorrt_llm8executor7versionEv"></span><span id="_CPPv2N12tensorrt_llm8executor7versionEv"></span><span id="tensorrt_llm::executor::version"></span><span class="target" id="executor_8h_1ada83e7ba49c4897a21cdb8706c97fce1"></span><span class="kt"><span class="pre">char</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">version</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7versionEv" title="Link to this definition"></a><br /></dt>
<dd><p>Version of TRT-LLM. </p>
</dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParamsE"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParamsE"></span><span id="tensorrt_llm::executor::ContextPhaseParams"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ContextPhaseParams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1aa81094bd70b775d6648abf5716a9251b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestIdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdType">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdType"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdType"></span><span id="tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams__VecTokens.RequestIdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a50ff5ae80ec9ca252934aa902c684a05"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ContextPhaseParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">firstGenTokens</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE" title="tensorrt_llm::executor::ContextPhaseParams::RequestIdType"><span class="n"><span class="pre">RequestIdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">reqId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdTypePv">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdTypePv"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdTypePv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams__VecTokens.RequestIdType.voidP"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a027257e6a680af913459d5ce2cb82b0c"></span><span class="sig-name descname"><span class="n"><span class="pre">ContextPhaseParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">firstGenTokens</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE" title="tensorrt_llm::executor::ContextPhaseParams::RequestIdType"><span class="n"><span class="pre">RequestIdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">reqId</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsE9VecTokens13RequestIdTypePv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERK18ContextPhaseParams">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERK18ContextPhaseParams"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERK18ContextPhaseParams"></span><span id="tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams__ContextPhaseParamsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a6656cde607b9fbe8c3810cce3364d69a"></span><span class="sig-name descname"><span class="n"><span class="pre">ContextPhaseParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERK18ContextPhaseParams" title="tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERK18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERR18ContextPhaseParams">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERR18ContextPhaseParams"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERR18ContextPhaseParams"></span><span id="tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams__ContextPhaseParamsRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1aceb361f1e38c515573ff87c0ce8b12c1"></span><span class="sig-name descname"><span class="n"><span class="pre">ContextPhaseParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERR18ContextPhaseParams" title="tensorrt_llm::executor::ContextPhaseParams::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams18ContextPhaseParamsERR18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsaSERK18ContextPhaseParams">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParamsaSERK18ContextPhaseParams"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParamsaSERK18ContextPhaseParams"></span><span id="tensorrt_llm::executor::ContextPhaseParams::assign-operator__ContextPhaseParamsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a8e835e9f16fd46d1caeff80f8720f634"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsaSERK18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsaSERR18ContextPhaseParams">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParamsaSERR18ContextPhaseParams"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParamsaSERR18ContextPhaseParams"></span><span id="tensorrt_llm::executor::ContextPhaseParams::assign-operator__ContextPhaseParamsRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a0d6ef185c183fb7d2ceeabcaf7ec0ac3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsaSERR18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParamsD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParamsD0Ev"></span><span id="tensorrt_llm::executor::ContextPhaseParams::~ContextPhaseParams"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a8ea9d3c141f9bbcb9639c8e1a630c57e"></span><span class="sig-name descname"><span class="n"><span class="pre">~ContextPhaseParams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18ContextPhaseParamseqERK18ContextPhaseParams">
<span id="_CPPv3NK12tensorrt_llm8executor18ContextPhaseParamseqERK18ContextPhaseParams"></span><span id="_CPPv2NK12tensorrt_llm8executor18ContextPhaseParamseqERK18ContextPhaseParams"></span><span id="tensorrt_llm::executor::ContextPhaseParams::eq-operator__ContextPhaseParamsCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a54426f8d39c0eab37287f3b313199caf"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18ContextPhaseParamseqERK18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NKR12tensorrt_llm8executor18ContextPhaseParams17getFirstGenTokensEv">
<span id="_CPPv3NKR12tensorrt_llm8executor18ContextPhaseParams17getFirstGenTokensEv"></span><span id="_CPPv2NKR12tensorrt_llm8executor18ContextPhaseParams17getFirstGenTokensEv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::getFirstGenTokensCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1acb2699e4cc7cb332c947c0827159b136"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getFirstGenTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NKR12tensorrt_llm8executor18ContextPhaseParams17getFirstGenTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NO12tensorrt_llm8executor18ContextPhaseParams17popFirstGenTokensEv">
<span id="_CPPv3NO12tensorrt_llm8executor18ContextPhaseParams17popFirstGenTokensEv"></span><span id="_CPPv2NO12tensorrt_llm8executor18ContextPhaseParams17popFirstGenTokensEv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::popFirstGenTokensO"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a71706e399ff075f0bea621300d1912d1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">popFirstGenTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">&amp;&amp;</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NO12tensorrt_llm8executor18ContextPhaseParams17popFirstGenTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18ContextPhaseParams8getReqIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor18ContextPhaseParams8getReqIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18ContextPhaseParams8getReqIdEv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::getReqIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1aad7f988239ce101c7bada72d204e8856"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE" title="tensorrt_llm::executor::ContextPhaseParams::RequestIdType"><span class="n"><span class="pre">RequestIdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getReqId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18ContextPhaseParams8getReqIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18ContextPhaseParams8getStateEv">
<span id="_CPPv3NK12tensorrt_llm8executor18ContextPhaseParams8getStateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18ContextPhaseParams8getStateEv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::getStateC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a66f6bf5b5bc42e4e0798e54e763e73ff"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getState</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18ContextPhaseParams8getStateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8getStateEv">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams8getStateEv"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams8getStateEv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::getState"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a80aa3ce0559fda9604398603a1153599"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getState</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8getStateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams12releaseStateEv">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams12releaseStateEv"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams12releaseStateEv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::releaseState"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1acb798adfd6c9643be6972307932e4959"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">releaseState</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams12releaseStateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8StatePtrE">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams8StatePtrE"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams8StatePtrE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1abb27654f82b7e3719f1383723bddbf49"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StatePtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="k"><span class="pre">decltype</span></span><span class="p"><span class="pre">(</span></span><span class="o"><span class="pre">&amp;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv" title="tensorrt_llm::executor::ContextPhaseParams::deleter"><span class="n"><span class="pre">deleter</span></span></a><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8StatePtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams6mReqIdE">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams6mReqIdE"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams6mReqIdE"></span><span id="tensorrt_llm::executor::ContextPhaseParams::mReqId__RequestIdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a948c936825dd49d70b437812c17ba03d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams13RequestIdTypeE" title="tensorrt_llm::executor::ContextPhaseParams::RequestIdType"><span class="n"><span class="pre">RequestIdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mReqId</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams6mReqIdE" title="Link to this definition"></a><br /></dt>
<dd><p>This request corresponds to the request ID in the context phase. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams15mFirstGenTokensE">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams15mFirstGenTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams15mFirstGenTokensE"></span><span id="tensorrt_llm::executor::ContextPhaseParams::mFirstGenTokens__VecTokens"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a05b2f569fee8bb174eb7a043bbeabb4d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFirstGenTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams15mFirstGenTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The first tokens generated by context executor. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams6mStateE">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams6mStateE"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams6mStateE"></span><span id="tensorrt_llm::executor::ContextPhaseParams::mState__StatePtr"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a3163863c98649154b4af531d0caad5f6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams8StatePtrE" title="tensorrt_llm::executor::ContextPhaseParams::StatePtr"><span class="n"><span class="pre">StatePtr</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mState</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="k"><span class="pre">nullptr</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv" title="tensorrt_llm::executor::ContextPhaseParams::deleter"><span class="n"><span class="pre">deleter</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams6mStateE" title="Link to this definition"></a><br /></dt>
<dd><p>Context phase state of this request. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv">
<span id="_CPPv3N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv"></span><span id="_CPPv2N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv"></span><span id="tensorrt_llm::executor::ContextPhaseParams::deleter__voidCP"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ContextPhaseParams_1a0d07065f40fffbdb98ec8d2aa350a739"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deleter</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParams7deleterEPKv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfigE">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfigE"></span><span id="tensorrt_llm::executor::DebugConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DebugConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for debugging output. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig11DebugConfigEbb9StringVec10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig11DebugConfigEbb9StringVec10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig11DebugConfigEbb9StringVec10SizeType32"></span><span id="tensorrt_llm::executor::DebugConfig::DebugConfig__b.b.StringVec.SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1a441df9531cf182fd1378ecbb0d92ecbc"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DebugConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">debugInputTensors</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">debugOutputTensors</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE" title="tensorrt_llm::executor::DebugConfig::StringVec"><span class="n"><span class="pre">StringVec</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">debugTensorNames</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">debugTensorsMaxIterations</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig11DebugConfigEbb9StringVec10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11DebugConfigeqERK11DebugConfig">
<span id="_CPPv3NK12tensorrt_llm8executor11DebugConfigeqERK11DebugConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor11DebugConfigeqERK11DebugConfig"></span><span id="tensorrt_llm::executor::DebugConfig::eq-operator__DebugConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1a8d9d2b7dd7685fb438931bc12a79c555"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfigeqERK11DebugConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11DebugConfig20getDebugInputTensorsEv">
<span id="_CPPv3NK12tensorrt_llm8executor11DebugConfig20getDebugInputTensorsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11DebugConfig20getDebugInputTensorsEv"></span><span id="tensorrt_llm::executor::DebugConfig::getDebugInputTensorsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1aff4941a872c0fb6025fe5c647a93d7ac"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDebugInputTensors</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig20getDebugInputTensorsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11DebugConfig21getDebugOutputTensorsEv">
<span id="_CPPv3NK12tensorrt_llm8executor11DebugConfig21getDebugOutputTensorsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11DebugConfig21getDebugOutputTensorsEv"></span><span id="tensorrt_llm::executor::DebugConfig::getDebugOutputTensorsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1afa176f8fd81f1f1f0d458880ceed890c"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDebugOutputTensors</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig21getDebugOutputTensorsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11DebugConfig19getDebugTensorNamesEv">
<span id="_CPPv3NK12tensorrt_llm8executor11DebugConfig19getDebugTensorNamesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11DebugConfig19getDebugTensorNamesEv"></span><span id="tensorrt_llm::executor::DebugConfig::getDebugTensorNamesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1ae3abfb3f9125c4771c012ddfa5e6343f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE" title="tensorrt_llm::executor::DebugConfig::StringVec"><span class="n"><span class="pre">StringVec</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getDebugTensorNames</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig19getDebugTensorNamesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11DebugConfig28getDebugTensorsMaxIterationsEv">
<span id="_CPPv3NK12tensorrt_llm8executor11DebugConfig28getDebugTensorsMaxIterationsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11DebugConfig28getDebugTensorsMaxIterationsEv"></span><span id="tensorrt_llm::executor::DebugConfig::getDebugTensorsMaxIterationsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1aa88d2b27e0fe9a82d79fed827a813a40"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDebugTensorsMaxIterations</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11DebugConfig28getDebugTensorsMaxIterationsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig20setDebugInputTensorsEb">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig20setDebugInputTensorsEb"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig20setDebugInputTensorsEb"></span><span id="tensorrt_llm::executor::DebugConfig::setDebugInputTensors__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1a89b3498cf45b85b70f71cd1ee9100ea1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDebugInputTensors</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">debugInputTensors</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig20setDebugInputTensorsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig21setDebugOutputTensorsEb">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig21setDebugOutputTensorsEb"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig21setDebugOutputTensorsEb"></span><span id="tensorrt_llm::executor::DebugConfig::setDebugOutputTensors__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1a965f04a1e84947f62135f8bed265cef1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDebugOutputTensors</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">debugOutputTensors</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig21setDebugOutputTensorsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig19setDebugTensorNamesERK9StringVec">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig19setDebugTensorNamesERK9StringVec"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig19setDebugTensorNamesERK9StringVec"></span><span id="tensorrt_llm::executor::DebugConfig::setDebugTensorNames__StringVecCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1a9f600e7172ec670e0cbe1e45f38b7148"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDebugTensorNames</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE" title="tensorrt_llm::executor::DebugConfig::StringVec"><span class="n"><span class="pre">StringVec</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">debugTensorNames</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig19setDebugTensorNamesERK9StringVec" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig28setDebugTensorsMaxIterationsE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig28setDebugTensorsMaxIterationsE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig28setDebugTensorsMaxIterationsE10SizeType32"></span><span id="tensorrt_llm::executor::DebugConfig::setDebugTensorsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1aa15dcb9f178e13549213315946c23e59"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDebugTensorsMaxIterations</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">debugTensorsMaxIterations</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig28setDebugTensorsMaxIterationsE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig9StringVecE"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig9StringVecE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1a477b51070633ac6ddff496518f5b0b59"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StringVec</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig18mDebugInputTensorsE">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig18mDebugInputTensorsE"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig18mDebugInputTensorsE"></span><span id="tensorrt_llm::executor::DebugConfig::mDebugInputTensors__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1aa03ddb48ba29e462a2c0f1fc74d7c2a7"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDebugInputTensors</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig18mDebugInputTensorsE" title="Link to this definition"></a><br /></dt>
<dd><p>If true, debug all input tensors. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig19mDebugOutputTensorsE">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig19mDebugOutputTensorsE"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig19mDebugOutputTensorsE"></span><span id="tensorrt_llm::executor::DebugConfig::mDebugOutputTensors__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1ad85f336e493ae2a557a41370333182cf"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDebugOutputTensors</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig19mDebugOutputTensorsE" title="Link to this definition"></a><br /></dt>
<dd><p>If true, debug all output tensors. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig17mDebugTensorNamesE">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig17mDebugTensorNamesE"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig17mDebugTensorNamesE"></span><span id="tensorrt_llm::executor::DebugConfig::mDebugTensorNames__StringVec"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1ae88247e3712e572334bcfef248b8d3e4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig9StringVecE" title="tensorrt_llm::executor::DebugConfig::StringVec"><span class="n"><span class="pre">StringVec</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDebugTensorNames</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig17mDebugTensorNamesE" title="Link to this definition"></a><br /></dt>
<dd><p>If not empty, only debug tensors in this list. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11DebugConfig26mDebugTensorsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor11DebugConfig26mDebugTensorsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor11DebugConfig26mDebugTensorsMaxIterationsE"></span><span id="tensorrt_llm::executor::DebugConfig::mDebugTensorsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DebugConfig_1abea452f5c201eb66e378f02960163f88"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDebugTensorsMaxIterations</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11DebugConfig26mDebugTensorsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd><p>If &gt; 0, provide debug tensors for at most debugTensorsMaxIterations past iterations, else dump them to files. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfigE"></span><span id="tensorrt_llm::executor::DecodingConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the decoding. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEENSt8optionalI11EagleConfigEE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEENSt8optionalI11EagleConfigEE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEENSt8optionalI11EagleConfigEE"></span><span id="tensorrt_llm::executor::DecodingConfig::DecodingConfig__std::optional:DecodingMode:.std::optional:LookaheadDecodingConfig:.std::optional:MedusaChoices:.std::optional:EagleConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1af6d3ff1a4fcb6a2114d587e055963e95"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decodingMode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">medusaChoices</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">eagleConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEENSt8optionalI11EagleConfigEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig"></span><span id="tensorrt_llm::executor::DecodingConfig::eq-operator__DecodingConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a72aae341b22cfafe152513cf5570c39f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode"></span><span id="tensorrt_llm::executor::DecodingConfig::setDecodingMode__DecodingModeCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a67e4a10d6f2bc039bd1a22d86a54b13b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDecodingMode</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode" title="Link to this definition"></a><br /></dt>
<dd><p>Sets decoding mode. Some modes require the use of their own setters. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getDecodingModeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1ad949398a15d2b85e99b569783243d069"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDecodingMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::DecodingConfig::setLookaheadDecoding__LookaheadDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a4e9181c201fcd72009a337bb706efc4f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLookaheadDecoding</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Sets lookahead decoding mode and config. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getLookaheadDecodingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a50e1dff2897795cf6a4bd5f0a4cc05b3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices"></span><span id="tensorrt_llm::executor::DecodingConfig::setMedusaChoices__MedusaChoicesCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1aa970dcb3b1ce45250e4f33febd6f20e8"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMedusaChoices</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices" title="Link to this definition"></a><br /></dt>
<dd><p>Sets medusa mode and config. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getMedusaChoicesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1afd3a11ae26135ac88540c5590de4e2f5"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMedusaChoices</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig14setEagleConfigERK11EagleConfig">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig14setEagleConfigERK11EagleConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig14setEagleConfigERK11EagleConfig"></span><span id="tensorrt_llm::executor::DecodingConfig::setEagleConfig__EagleConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a816f97fda3245071c255534ee020a47f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEagleConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14setEagleConfigERK11EagleConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Sets eagle mode and config. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig14getEagleConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig14getEagleConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig14getEagleConfigEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getEagleConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a019c2bcf78f01730757d50af7ba9ba08"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEagleConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig14getEagleConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE"></span><span id="tensorrt_llm::executor::DecodingConfig::mDecodingMode__std::optional:DecodingMode:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a44b33dcb361289657af45112b7d7ef65"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE"></span><span id="tensorrt_llm::executor::DecodingConfig::mLookaheadDecodingConfig__std::optional:LookaheadDecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a714a527e462727b8aba4c8ff4e18c35e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLookaheadDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE"></span><span id="tensorrt_llm::executor::DecodingConfig::mMedusaChoices__std::optional:MedusaChoices:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a13bb97024510b32cbb56b1606742e7ec"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMedusaChoices</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig12mEagleConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig12mEagleConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig12mEagleConfigE"></span><span id="tensorrt_llm::executor::DecodingConfig::mEagleConfig__std::optional:EagleConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1ac03e1f2278412e56fddd447cde2bae56"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEagleConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig12mEagleConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfigE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DynamicBatchConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for dynamic tuning of batch size and max num tokens. During runtime the statistics of input and output lengths are recoreded. Based on these statistics, the batch size and max num tokens are tuned dynamically to better serve the requests. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig18DynamicBatchConfigEbb10SizeType32NSt6vectorINSt4pairI10SizeType3210SizeType32EEEE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig18DynamicBatchConfigEbb10SizeType32NSt6vectorINSt4pairI10SizeType3210SizeType32EEEE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig18DynamicBatchConfigEbb10SizeType32NSt6vectorINSt4pairI10SizeType3210SizeType32EEEE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::DynamicBatchConfig__b.b.SizeType32.std::vector:std::pair:SizeType32.SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a5a375564bc43dafa4335a84eb8efbe41"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DynamicBatchConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableBatchSizeTuning</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableMaxNumTokensTuning</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dynamicBatchMovingAverageWindow</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig39kDefaultDynamicBatchMovingAverageWindowE" title="tensorrt_llm::executor::DynamicBatchConfig::kDefaultDynamicBatchMovingAverageWindow"><span class="n"><span class="pre">kDefaultDynamicBatchMovingAverageWindow</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">pair</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">batchSizeTable</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22kDefaultBatchSizeTableE" title="tensorrt_llm::executor::DynamicBatchConfig::kDefaultBatchSizeTable"><span class="n"><span class="pre">kDefaultBatchSizeTable</span></span></a><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig18DynamicBatchConfigEbb10SizeType32NSt6vectorINSt4pairI10SizeType3210SizeType32EEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig34getDynamicBatchMovingAverageWindowEv">
<span id="_CPPv3NK12tensorrt_llm8executor18DynamicBatchConfig34getDynamicBatchMovingAverageWindowEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18DynamicBatchConfig34getDynamicBatchMovingAverageWindowEv"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::getDynamicBatchMovingAverageWindowC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1ace3684b897f4e1918c7c16f7dbb0f567"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDynamicBatchMovingAverageWindow</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig34getDynamicBatchMovingAverageWindowEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig24getEnableBatchSizeTuningEv">
<span id="_CPPv3NK12tensorrt_llm8executor18DynamicBatchConfig24getEnableBatchSizeTuningEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18DynamicBatchConfig24getEnableBatchSizeTuningEv"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::getEnableBatchSizeTuningC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1ae1333e1c22722d1c6f6873bbf8723b73"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableBatchSizeTuning</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig24getEnableBatchSizeTuningEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig27getEnableMaxNumTokensTuningEv">
<span id="_CPPv3NK12tensorrt_llm8executor18DynamicBatchConfig27getEnableMaxNumTokensTuningEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18DynamicBatchConfig27getEnableMaxNumTokensTuningEv"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::getEnableMaxNumTokensTuningC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1ab0086c1cfad34346cd69749ae2ac753f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableMaxNumTokensTuning</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig27getEnableMaxNumTokensTuningEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig17getBatchSizeTableEv">
<span id="_CPPv3NK12tensorrt_llm8executor18DynamicBatchConfig17getBatchSizeTableEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18DynamicBatchConfig17getBatchSizeTableEv"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::getBatchSizeTableC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a874f07528531eff13ae54b28daf1b49b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">pair</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBatchSizeTable</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18DynamicBatchConfig17getBatchSizeTableEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig39kDefaultDynamicBatchMovingAverageWindowE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig39kDefaultDynamicBatchMovingAverageWindowE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig39kDefaultDynamicBatchMovingAverageWindowE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::kDefaultDynamicBatchMovingAverageWindow__SizeType32C"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a384341c4c7bd1d935ec62852cb10290d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultDynamicBatchMovingAverageWindow</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">128</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig39kDefaultDynamicBatchMovingAverageWindowE" title="Link to this definition"></a><br /></dt>
<dd><p>The default window size for moving average of input and output length which is used to calculate dynamic batch size and max num tokens. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22kDefaultBatchSizeTableE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig22kDefaultBatchSizeTableE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig22kDefaultBatchSizeTableE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::kDefaultBatchSizeTable__std::vector:std::pair:SizeType32.SizeType32::C"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a2552da34e844c7cbfcd9c4dd2fc38a7d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">pair</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultBatchSizeTable</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22kDefaultBatchSizeTableE" title="Link to this definition"></a><br /></dt>
<dd><p>The default value of batch size table. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22mEnableBatchSizeTuningE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig22mEnableBatchSizeTuningE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig22mEnableBatchSizeTuningE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::mEnableBatchSizeTuning__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a90b2eb7cff53639b2e35846ec27c771f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableBatchSizeTuning</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig22mEnableBatchSizeTuningE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if the batch size should be tuned dynamically. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig25mEnableMaxNumTokensTuningE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig25mEnableMaxNumTokensTuningE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig25mEnableMaxNumTokensTuningE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::mEnableMaxNumTokensTuning__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a1a0e19a1020c36566d642eb0d5ec3a36"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableMaxNumTokensTuning</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig25mEnableMaxNumTokensTuningE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if the max num tokens should be tuned dynamically. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig32mDynamicBatchMovingAverageWindowE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig32mDynamicBatchMovingAverageWindowE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig32mDynamicBatchMovingAverageWindowE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::mDynamicBatchMovingAverageWindow__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a24b17e19c808eeabe3263296535d12b5"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDynamicBatchMovingAverageWindow</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig32mDynamicBatchMovingAverageWindowE" title="Link to this definition"></a><br /></dt>
<dd><p>The window size for moving average of input and output length which is used to calculate dynamic batch size and max num tokens. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig15mBatchSizeTableE">
<span id="_CPPv3N12tensorrt_llm8executor18DynamicBatchConfig15mBatchSizeTableE"></span><span id="_CPPv2N12tensorrt_llm8executor18DynamicBatchConfig15mBatchSizeTableE"></span><span id="tensorrt_llm::executor::DynamicBatchConfig::mBatchSizeTable__std::vector:std::pair:SizeType32.SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DynamicBatchConfig_1a4050c896c0b2ac162d4745117ebb56a0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">pair</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBatchSizeTable</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfig15mBatchSizeTableE" title="Link to this definition"></a><br /></dt>
<dd><p>A vector of (batchSizeLimit, batchSize). When max capacity batch size is less than. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11EagleConfigE">
<span id="_CPPv3N12tensorrt_llm8executor11EagleConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor11EagleConfigE"></span><span id="tensorrt_llm::executor::EagleConfig"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">EagleConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11EagleConfig11EagleConfigENSt8optionalI12EagleChoicesEEbNSt8optionalIfEE">
<span id="_CPPv3N12tensorrt_llm8executor11EagleConfig11EagleConfigENSt8optionalI12EagleChoicesEEbNSt8optionalIfEE"></span><span id="_CPPv2N12tensorrt_llm8executor11EagleConfig11EagleConfigENSt8optionalI12EagleChoicesEEbNSt8optionalIfEE"></span><span id="tensorrt_llm::executor::EagleConfig::EagleConfig__std::optional:EagleChoices:.b.std::optional:float:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1ac7222df59373601695882b45f4cb52d8"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">EagleConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12EagleChoicesE" title="tensorrt_llm::executor::EagleChoices"><span class="n"><span class="pre">EagleChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">eagleChoices</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">greedySampling</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">posteriorThreshold</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig11EagleConfigENSt8optionalI12EagleChoicesEEbNSt8optionalIfEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11EagleConfigeqERK11EagleConfig">
<span id="_CPPv3NK12tensorrt_llm8executor11EagleConfigeqERK11EagleConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor11EagleConfigeqERK11EagleConfig"></span><span id="tensorrt_llm::executor::EagleConfig::eq-operator__EagleConfigCRC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1a911ae6f733aa6621787773db6875c888"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfigeqERK11EagleConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11EagleConfig15getEagleChoicesEv">
<span id="_CPPv3NK12tensorrt_llm8executor11EagleConfig15getEagleChoicesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11EagleConfig15getEagleChoicesEv"></span><span id="tensorrt_llm::executor::EagleConfig::getEagleChoicesC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1ada639d3d66b9a05b5a8804c39906c3f9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12EagleChoicesE" title="tensorrt_llm::executor::EagleChoices"><span class="n"><span class="pre">EagleChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEagleChoices</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfig15getEagleChoicesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11EagleConfig21getPosteriorThresholdEv">
<span id="_CPPv3NK12tensorrt_llm8executor11EagleConfig21getPosteriorThresholdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11EagleConfig21getPosteriorThresholdEv"></span><span id="tensorrt_llm::executor::EagleConfig::getPosteriorThresholdC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1a39be08f82dd0b5387bc3ebd751b93084"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPosteriorThreshold</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfig21getPosteriorThresholdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11EagleConfig16isGreedySamplingEv">
<span id="_CPPv3NK12tensorrt_llm8executor11EagleConfig16isGreedySamplingEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11EagleConfig16isGreedySamplingEv"></span><span id="tensorrt_llm::executor::EagleConfig::isGreedySamplingC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1a0bebe607080f2e6dbe6835df155dc4b3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isGreedySampling</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11EagleConfig16isGreedySamplingEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11EagleConfig19checkPosteriorValueERKNSt8optionalIfEE">
<span id="_CPPv3N12tensorrt_llm8executor11EagleConfig19checkPosteriorValueERKNSt8optionalIfEE"></span><span id="_CPPv2N12tensorrt_llm8executor11EagleConfig19checkPosteriorValueERKNSt8optionalIfEE"></span><span id="tensorrt_llm::executor::EagleConfig::checkPosteriorValue__std::optional:float:CR"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1acc18740b06d0c59ba7481a4a8e549f3e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkPosteriorValue</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">value</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig19checkPosteriorValueERKNSt8optionalIfEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11EagleConfig13mEagleChoicesE">
<span id="_CPPv3N12tensorrt_llm8executor11EagleConfig13mEagleChoicesE"></span><span id="_CPPv2N12tensorrt_llm8executor11EagleConfig13mEagleChoicesE"></span><span id="tensorrt_llm::executor::EagleConfig::mEagleChoices__std::optional:EagleChoices:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1afb9fa382408155bdafb7a611d37b7931"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12EagleChoicesE" title="tensorrt_llm::executor::EagleChoices"><span class="n"><span class="pre">EagleChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEagleChoices</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig13mEagleChoicesE" title="Link to this definition"></a><br /></dt>
<dd><p>choices forming tree for EAGLE-1. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11EagleConfig15mGreedySamplingE">
<span id="_CPPv3N12tensorrt_llm8executor11EagleConfig15mGreedySamplingE"></span><span id="_CPPv2N12tensorrt_llm8executor11EagleConfig15mGreedySamplingE"></span><span id="tensorrt_llm::executor::EagleConfig::mGreedySampling__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1a36836b50046667b4ca54bb120aaa1998"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGreedySampling</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig15mGreedySamplingE" title="Link to this definition"></a><br /></dt>
<dd><p>Flag to use greedy or typical acceptance. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11EagleConfig19mPosteriorThresholdE">
<span id="_CPPv3N12tensorrt_llm8executor11EagleConfig19mPosteriorThresholdE"></span><span id="_CPPv2N12tensorrt_llm8executor11EagleConfig19mPosteriorThresholdE"></span><span id="tensorrt_llm::executor::EagleConfig::mPosteriorThreshold__std::optional:float:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1EagleConfig_1ad26ec15a330911822577a7a483b0a1d7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPosteriorThreshold</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11EagleConfig19mPosteriorThresholdE" title="Link to this definition"></a><br /></dt>
<dd><p>Minimum token probability of the typical acceptance. Corresponds to epsilon in <a class="reference external" href="https://arxiv.org/pdf/2401.10774">https://arxiv.org/pdf/2401.10774</a>. Default is 0.09f. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ExecutorE">
<span id="_CPPv3N12tensorrt_llm8executor8ExecutorE"></span><span id="_CPPv2N12tensorrt_llm8executor8ExecutorE"></span><span id="tensorrt_llm::executor::Executor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>The executor is responsible for receiving new requests and sending responses, and running the inference. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::filesystem::pathCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a74e112aaa1be8f0b62f8d96f59021e1a"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelPath</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>modelPath</strong> – Path to the folder that defines the model to run </p></li>
<li><p><strong>modelType</strong> – The type of model </p></li>
<li><p><strong>executorConfig</strong> – The configuration for the executor </p></li>
<li><p><strong>comm</strong> – An optional inter-process communicator configuration </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::filesystem::pathCR.std::filesystem::pathCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af2fb4913e74a0cc0b3a85ef58bd1f662"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderModelPath</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decoderModelPath</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfigRKNSt8optionalINSt3mapINSt6stringE6TensorEEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfigRKNSt8optionalINSt3mapINSt6stringE6TensorEEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfigRKNSt8optionalINSt3mapINSt6stringE6TensorEEEE"></span><span id="tensorrt_llm::executor::Executor::Executor__BufferViewCR.ssCR.ModelType.ExecutorConfigCR.std::optional:std::map:ss.Tensor::CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ab16c36317e9f3389f6d6ca0490e892d9"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BufferViewE" title="tensorrt_llm::executor::BufferView"><span class="n"><span class="pre">BufferView</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">engineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">jsonConfigStr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">map</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">managedWeights</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfigRKNSt8optionalINSt3mapINSt6stringE6TensorEEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__BufferViewCR.ssCR.BufferViewCR.ssCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af79e2bea60216502ba77f33508327300"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BufferViewE" title="tensorrt_llm::executor::BufferView"><span class="n"><span class="pre">BufferView</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderEngineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderJsonConfigStr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BufferViewE" title="tensorrt_llm::executor::BufferView"><span class="n"><span class="pre">BufferView</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decoderEngineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decoderJsonConfigStr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK10BufferViewRKNSt6stringERK10BufferViewRKNSt6stringE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::shared_ptr:Model:.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a9a98e9c79f95e75c2a16f900986082ce"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Model</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">model</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::shared_ptr:Model:.std::shared_ptr:Model:.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1aedb3a745955f57fa268a326e49c0f741"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Model</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encoderModel</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Model</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decoderModel</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ExecutorD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor8ExecutorD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor8ExecutorD0Ev"></span><span id="tensorrt_llm::executor::Executor::~Executor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a7c5a64a1cc125f8b6a17f05c85530b57"></span><span class="sig-name descname"><span class="n"><span class="pre">~Executor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ExecutorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK8Executor">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERK8Executor"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERK8Executor"></span><span id="tensorrt_llm::executor::Executor::Executor__ExecutorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af5a2778fb124f2e594c2f6e30a6e8322"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK8Executor" title="tensorrt_llm::executor::Executor::Executor"><span class="n"><span class="pre">Executor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executor</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERK8Executor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ExecutoraSERK8Executor">
<span id="_CPPv3N12tensorrt_llm8executor8ExecutoraSERK8Executor"></span><span id="_CPPv2N12tensorrt_llm8executor8ExecutoraSERK8Executor"></span><span id="tensorrt_llm::executor::Executor::assign-operator__ExecutorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a375b8e9c2ec8f50865044eea22f949fb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE" title="tensorrt_llm::executor::Executor"><span class="n"><span class="pre">Executor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE" title="tensorrt_llm::executor::Executor"><span class="n"><span class="pre">Executor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executor</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">delete</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ExecutoraSERK8Executor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERR8Executor">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERR8Executor"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERR8Executor"></span><span id="tensorrt_llm::executor::Executor::Executor__ExecutorRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a5d8813384914105ab93bc5afc6ef633e"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERR8Executor" title="tensorrt_llm::executor::Executor::Executor"><span class="n"><span class="pre">Executor</span></span></a><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERR8Executor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ExecutoraSERR8Executor">
<span id="_CPPv3N12tensorrt_llm8executor8ExecutoraSERR8Executor"></span><span id="_CPPv2N12tensorrt_llm8executor8ExecutoraSERR8Executor"></span><span id="tensorrt_llm::executor::Executor::assign-operator__ExecutorRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ab35c6ac71a9d0846232caebcd39db3ad"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE" title="tensorrt_llm::executor::Executor"><span class="n"><span class="pre">Executor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE" title="tensorrt_llm::executor::Executor"><span class="n"><span class="pre">Executor</span></span></a><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ExecutoraSERR8Executor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request"></span><span id="tensorrt_llm::executor::Executor::enqueueRequest__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af760463b7371a09773207283f9f0fbb6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">enqueueRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request" title="Link to this definition"></a><br /></dt>
<dd><p>Enqueue a new request. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>request</strong> – The LLM request which contains input tokens and request parameters </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A unique id that identifies the request </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE"></span><span id="tensorrt_llm::executor::Executor::enqueueRequests__std::vector:Request:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af9acbedea80c9d77d8c77d9f66a74adb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">enqueueRequests</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requests</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE" title="Link to this definition"></a><br /></dt>
<dd><p>Enqueue a batch of request. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::Executor::awaitResponses__std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a163139b9705933beade11af62468f127"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for ready responses. </p>
<p><div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">This</span> <span class="n">overload</span> <span class="n">awaits</span> <span class="k">for</span> <span class="nb">any</span> <span class="n">ready</span> <span class="n">responses</span><span class="o">.</span> <span class="n">In</span> <span class="n">particular</span><span class="p">,</span> <span class="k">if</span> <span class="n">several</span> <span class="n">requests</span>
<span class="n">have</span> <span class="n">been</span> <span class="n">enqueued</span><span class="p">,</span> <span class="n">this</span> <span class="n">method</span> <span class="n">will</span> <span class="n">provide</span> <span class="nb">any</span> <span class="n">ready</span> <span class="n">responses</span> <span class="n">without</span> <span class="n">order</span> <span class="n">guarantees</span><span class="o">.</span>
</pre></div>
</div>
</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>timeout</strong> – The maximum time to wait for new responses </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::Executor::awaitResponses__IdTypeCR.std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ab08f8c1d216d8e9f9c8a214d72d69f65"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitResponses</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestId</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for ready responses. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>id</strong> – A request id </p></li>
<li><p><strong>timeout</strong> – The maximum time to wait for new responses </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::Executor::awaitResponses__std::vector:IdType:CR.std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a8954fd92f602eab2467de3f1cd419661"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestIds</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for multiple ready responses. </p>
<p><div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">A</span> <span class="n">multiple</span> <span class="n">ID</span> <span class="n">request</span> <span class="n">behaves</span> <span class="k">as</span> <span class="k">if</span> <span class="n">awaitResponses</span><span class="p">(</span><span class="n">IdType</span><span class="p">,</span> <span class="n">timeout</span><span class="p">)</span>
<span class="n">were</span> <span class="n">invoked</span> <span class="n">on</span> <span class="nb">all</span> <span class="n">IDs</span><span class="o">.</span> <span class="n">The</span> <span class="n">returned</span> <span class="n">vector</span> <span class="n">contains</span>
<span class="n">a</span> <span class="n">vector</span> <span class="n">of</span> <span class="n">responses</span> <span class="n">per</span> <span class="n">ID</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">same</span> <span class="n">order</span> <span class="n">specified</span> <span class="n">by</span> <span class="n">the</span> <span class="n">requestIds</span><span class="o">.</span>
<span class="n">The</span> <span class="n">same</span> <span class="n">behaviour</span> <span class="k">as</span> <span class="n">awaitResponses</span><span class="p">(</span><span class="n">IdType</span><span class="p">,</span> <span class="n">timeout</span><span class="p">)</span> <span class="n">applies</span><span class="p">:</span>
<span class="o">*</span> <span class="n">Responses</span> <span class="n">may</span> <span class="n">be</span> <span class="n">empty</span><span class="o">.</span>
<span class="o">*</span> <span class="n">If</span> <span class="nb">all</span> <span class="n">responses</span> <span class="n">have</span> <span class="n">already</span> <span class="n">been</span> <span class="n">given</span> <span class="k">for</span> <span class="n">one</span> <span class="n">of</span> <span class="n">the</span> <span class="n">requestIds</span><span class="p">,</span>
<span class="n">then</span> <span class="n">this</span> <span class="n">method</span> <span class="n">will</span> <span class="n">hang</span> <span class="n">unless</span> <span class="n">a</span> <span class="n">timeout</span> <span class="ow">is</span> <span class="n">specified</span><span class="o">.</span>
</pre></div>
</div>
</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>requestIds</strong> – Ids requested </p></li>
<li><p><strong>timeout</strong> – The maximum time to wait for new responses </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of vector of responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE">
<span id="_CPPv3NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE"></span><span id="_CPPv2NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE"></span><span id="tensorrt_llm::executor::Executor::getNumResponsesReady__std::optional:IdType:CRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a8870564de9e1b838e3a24e770de65e97"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumResponsesReady</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE" title="Link to this definition"></a><br /></dt>
<dd><p>Get the number of ready responses. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>requestId</strong> – An optional request id </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>The number of ready responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor13cancelRequestE6IdType">
<span id="_CPPv3N12tensorrt_llm8executor8Executor13cancelRequestE6IdType"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor13cancelRequestE6IdType"></span><span id="tensorrt_llm::executor::Executor::cancelRequest__IdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ad157e5c600e0bcd1fb16695d573ec279"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cancelRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor13cancelRequestE6IdType" title="Link to this definition"></a><br /></dt>
<dd><p>Cancel the request with provided request id. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>id</strong> – The request id for which to cancel the response </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8shutdownEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8shutdownEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8shutdownEv"></span><span id="tensorrt_llm::executor::Executor::shutdown"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1abee0bdae43e18a83ada60f4e63eb72d5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shutdown</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8shutdownEv" title="Link to this definition"></a><br /></dt>
<dd><p>Signals the server to shutdown. </p>
<p>This call is blocking. Only returns when all requests have terminated or timeout has been reached </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv"></span><span id="tensorrt_llm::executor::Executor::getLatestIterationStats"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a86d2af2924046fdff9fb70a9e86159bd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">deque</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLatestIterationStats</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the per-iterations statistics computed since last call to getLatestIterationStats. Contains at most iterStatsMaxIterations iterations. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>Iteration stats </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv"></span><span id="tensorrt_llm::executor::Executor::getLatestRequestStats"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ab75ad0e4223d528aa87fdd18d1caeec0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">deque</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLatestRequestStats</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the request stats of each iteration computed since last call to getLatestRequestStats. Contains at most requestStatsMaxIterations iterations. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p><a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> stats grouped by iterations </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor21getLatestDebugTensorsEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor21getLatestDebugTensorsEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor21getLatestDebugTensorsEv"></span><span id="tensorrt_llm::executor::Executor::getLatestDebugTensors"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a65398075a94e7065f0c5a0d7ae4e4538"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">deque</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIterationE" title="tensorrt_llm::executor::DebugTensorsPerIteration"><span class="n"><span class="pre">DebugTensorsPerIteration</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLatestDebugTensors</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor21getLatestDebugTensorsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the debug tensors of each iteration computed since last call to getLatestDebugTensors. Contains at most debugTensorsMaxIterations iterations. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p><a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> debug tensors grouped by iterations </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv"></span><span id="tensorrt_llm::executor::Executor::canEnqueueRequestsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a385dcd6e4c171cc8dcc6b7e327a8524b"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">canEnqueueRequests</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if the current process is allowed to enqueueRequests. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Executor13isParticipantEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Executor13isParticipantEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Executor13isParticipantEv"></span><span id="tensorrt_llm::executor::Executor::isParticipantC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a2b6ddd22f24a00cf5eeaafce158aaf64"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isParticipant</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Executor13isParticipantEv" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if the current process participates in this executor instance. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Executor22getKVCacheEventManagerEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Executor22getKVCacheEventManagerEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Executor22getKVCacheEventManagerEv"></span><span id="tensorrt_llm::executor::Executor::getKVCacheEventManagerC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a4c21f8a3611d700e1757ac2fb4348836"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManagerE" title="tensorrt_llm::executor::KVCacheEventManager"><span class="n"><span class="pre">KVCacheEventManager</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getKVCacheEventManager</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Executor22getKVCacheEventManagerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor5mImplE"></span><span id="tensorrt_llm::executor::Executor::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a4b7affd37a154513db81d23acb941801"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExecutorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the model executor. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType3215SchedulerConfig13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI25LogitsPostProcessorConfigEENSt8optionalI14DecodingConfigEEfNSt8optionalI10SizeType32EERK29ExtendedRuntimePerfKnobConfigNSt8optionalI11DebugConfigEE10SizeType328uint64_tNSt8optionalI25SpeculativeDecodingConfigEENSt8optionalI20GuidedDecodingConfigEE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType3215SchedulerConfig13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI25LogitsPostProcessorConfigEENSt8optionalI14DecodingConfigEEfNSt8optionalI10SizeType32EERK29ExtendedRuntimePerfKnobConfigNSt8optionalI11DebugConfigEE10SizeType328uint64_tNSt8optionalI25SpeculativeDecodingConfigEENSt8optionalI20GuidedDecodingConfigEE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType3215SchedulerConfig13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI25LogitsPostProcessorConfigEENSt8optionalI14DecodingConfigEEfNSt8optionalI10SizeType32EERK29ExtendedRuntimePerfKnobConfigNSt8optionalI11DebugConfigEE10SizeType328uint64_tNSt8optionalI25SpeculativeDecodingConfigEENSt8optionalI20GuidedDecodingConfigEE"></span><span id="tensorrt_llm::executor::ExecutorConfig::ExecutorConfig__SizeType32.SchedulerConfig.KvCacheConfig.b.b.SizeType32.SizeType32.BatchingType.std::optional:SizeType32:.std::optional:SizeType32:.std::optional:ParallelConfig:.std::optional:PeftCacheConfig:CR.std::optional:LogitsPostProcessorConfig:.std::optional:DecodingConfig:.float.std::optional:SizeType32:.ExtendedRuntimePerfKnobConfigCR.std::optional:DebugConfig:.SizeType32.uint64_t.std::optional:SpeculativeDecodingConfig:.std::optional:GuidedDecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac711bf39afe96a2689423ad4229f1bcd"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExecutorConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">schedulerConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableChunkedContext</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">normalizeLogProbs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">iterStatsMaxIterations</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultIterStatsMaxIterationsE" title="tensorrt_llm::executor::ExecutorConfig::kDefaultIterStatsMaxIterations"><span class="n"><span class="pre">kDefaultIterStatsMaxIterations</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestStatsMaxIterations</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig33kDefaultRequestStatsMaxIterationsE" title="tensorrt_llm::executor::ExecutorConfig::kDefaultRequestStatsMaxIterations"><span class="n"><span class="pre">kDefaultRequestStatsMaxIterations</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchingType</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE" title="tensorrt_llm::executor::BatchingType::kINFLIGHT"><span class="n"><span class="pre">kINFLIGHT</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">parallelConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE" title="tensorrt_llm::executor::LogitsPostProcessorConfig"><span class="n"><span class="pre">LogitsPostProcessorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsPostProcessorConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decodingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">gpuWeightsPercent</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxQueueSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">extendedRuntimePerfKnobConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">debugConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">recvPollPeriodMs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <span class="n"><span class="pre">uint64_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxSeqIdleMicroseconds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultMaxSeqIdleMicrosecondsE" title="tensorrt_llm::executor::ExecutorConfig::kDefaultMaxSeqIdleMicroseconds"><span class="n"><span class="pre">kDefaultMaxSeqIdleMicroseconds</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">specDecConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">guidedDecodingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType3215SchedulerConfig13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI25LogitsPostProcessorConfigEENSt8optionalI14DecodingConfigEEfNSt8optionalI10SizeType32EERK29ExtendedRuntimePerfKnobConfigNSt8optionalI11DebugConfigEE10SizeType328uint64_tNSt8optionalI25SpeculativeDecodingConfigEENSt8optionalI20GuidedDecodingConfigEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxBeamWidthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af28528ea0a4f5d8252d5cf4aa809cab9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxBeamWidth</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getSchedulerConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ad04adf7773f5dc011df7811ed71a2065"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSchedulerConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getKvCacheConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a81554cd8e28a7c975dcb88e0bc678829"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getKvCacheConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig21getSchedulerConfigRefEv">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig21getSchedulerConfigRefEv"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig21getSchedulerConfigRefEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getSchedulerConfigRef"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac21502045320dd9d10638451f7a75488"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getSchedulerConfigRef</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21getSchedulerConfigRefEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig19getKvCacheConfigRefEv">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig19getKvCacheConfigRefEv"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig19getKvCacheConfigRefEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getKvCacheConfigRef"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a71c8f73c26ef6e876eefdc78128c4b5d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getKvCacheConfigRef</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig19getKvCacheConfigRefEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getEnableChunkedContextC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acebbdef0320ea601a14d69ee84b0d8e8"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableChunkedContext</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getNormalizeLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af9638fec77c9ad7a4a9230c6b8ccd3e2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNormalizeLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getIterStatsMaxIterationsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ad68d1ec95040a53535b2df4bbc5e67ad"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getIterStatsMaxIterations</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getRequestStatsMaxIterationsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a80fbe7fc988029b4e5605bc79d8983e5"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRequestStatsMaxIterations</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getBatchingTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac90fe84eae462d1eae8d2ad84c4c8bee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBatchingType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxBatchSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a602a24b9c115d92f33b59045a9f9dbd7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxBatchSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxNumTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a201dbdac3a1cdd40d766f966f42905c4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxNumTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getParallelConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a301d8dae1d5715022c95eda417db6c6e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParallelConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getPeftCacheConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a6e6e228a04ba55d677e1903c24ca9cb0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPeftCacheConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getLogitsPostProcessorConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig28getLogitsPostProcessorConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig28getLogitsPostProcessorConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getLogitsPostProcessorConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a442a13ce5bc822eda741b7677d0b6709"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE" title="tensorrt_llm::executor::LogitsPostProcessorConfig"><span class="n"><span class="pre">LogitsPostProcessorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogitsPostProcessorConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getLogitsPostProcessorConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getDecodingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a3b3bc02cbbf6c525d2643c0206409785"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getGpuWeightsPercentC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a45340052913aba3d6e3acdf12b07d4fd"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpuWeightsPercent</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxQueueSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxQueueSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxQueueSizeEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxQueueSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7a79bd90a5dee83095bd3c6596d4b463"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxQueueSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxQueueSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig32getExtendedRuntimePerfKnobConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig32getExtendedRuntimePerfKnobConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig32getExtendedRuntimePerfKnobConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getExtendedRuntimePerfKnobConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7713e188a26de4fb490ad99a4eb385e6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getExtendedRuntimePerfKnobConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig32getExtendedRuntimePerfKnobConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig14getDebugConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig14getDebugConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig14getDebugConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getDebugConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a412b3fcaeb31bdeddfe73a1be474d450"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDebugConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig14getDebugConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig19getRecvPollPeriodMsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig19getRecvPollPeriodMsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig19getRecvPollPeriodMsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getRecvPollPeriodMsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a8f75db7250ff15d1255a56915234cb35"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRecvPollPeriodMs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig19getRecvPollPeriodMsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getMaxSeqIdleMicrosecondsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig25getMaxSeqIdleMicrosecondsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig25getMaxSeqIdleMicrosecondsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxSeqIdleMicrosecondsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7a9d21d7e4b3aa5e0ae43011f520d668"></span><span class="n"><span class="pre">uint64_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxSeqIdleMicroseconds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getMaxSeqIdleMicrosecondsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getSpecDecConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig16getSpecDecConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig16getSpecDecConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getSpecDecConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a704988c0c750a3321f0c89a5477914dd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSpecDecConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getSpecDecConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getGuidedDecodingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig23getGuidedDecodingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig23getGuidedDecodingConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getGuidedDecodingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a4fe6cf353990885c3eb41bef8d6a7e80"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGuidedDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getGuidedDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a4a9426ca428d09d9514e85a0c2b612f5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxBeamWidth</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxBatchSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acbccae1669e096e7214a532e8ca415dc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxBatchSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxNumTokens__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af273f0c3f408e7b54354b00a9d69ca3a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxNumTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setSchedulerConfig__SchedulerConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a3d0b81d8422cbb579ba37d425c600004"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSchedulerConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setKvCacheConfig__KvCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1abdfb27a2cc58da0a28ce5c2486ea688f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setKvCacheConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb"></span><span id="tensorrt_llm::executor::ExecutorConfig::setEnableChunkedContext__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a6426dbe445f65be08fae30ddc03135f4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEnableChunkedContext</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableChunkedContext</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb"></span><span id="tensorrt_llm::executor::ExecutorConfig::setNormalizeLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aa40eff9c23c0abd6d4be7977bb80d80a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNormalizeLogProbs</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">normalizeLogProbs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setIterStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acab1aa5f729a9d145d23c0a0c8c2e526"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setIterStatsMaxIterations</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">iterStatsMaxIterations</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setRequestStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acbb1bfe2c8a77c80d3ec6e55ff00196a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRequestStatsMaxIterations</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestStatsMaxIterations</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType"></span><span id="tensorrt_llm::executor::ExecutorConfig::setBatchingType__BatchingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7171a21854120d745c185d6d2ecd9673"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBatchingType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchingType</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setParallelConfig__ParallelConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a36697066f6b8a8e7ea8d031ce1bfdaca"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setParallelConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">parallelConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setPeftCacheConfig__PeftCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af5453098e1af9097cc47acb28d1df39a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPeftCacheConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setLogitsPostProcessorConfigERK25LogitsPostProcessorConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig28setLogitsPostProcessorConfigERK25LogitsPostProcessorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig28setLogitsPostProcessorConfigERK25LogitsPostProcessorConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setLogitsPostProcessorConfig__LogitsPostProcessorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aabc5cd613e383abd181d7b8e6347a1d2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLogitsPostProcessorConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE" title="tensorrt_llm::executor::LogitsPostProcessorConfig"><span class="n"><span class="pre">LogitsPostProcessorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logitsPostProcessorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setLogitsPostProcessorConfigERK25LogitsPostProcessorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setDecodingConfig__DecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7d6173ec1a87d222ff1674d047b3d69b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDecodingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf"></span><span id="tensorrt_llm::executor::ExecutorConfig::setGpuWeightsPercent__floatCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a555f875ac320973a0e0bc94e0e9611b1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setGpuWeightsPercent</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">gpuWeightsPercent</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxQueueSizeERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxQueueSizeERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxQueueSizeERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxQueueSize__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ad3ae79600c15c02fd14ddb5642ed0455"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxQueueSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">maxQueueSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxQueueSizeERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig32setExtendedRuntimePerfKnobConfigERK29ExtendedRuntimePerfKnobConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig32setExtendedRuntimePerfKnobConfigERK29ExtendedRuntimePerfKnobConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig32setExtendedRuntimePerfKnobConfigERK29ExtendedRuntimePerfKnobConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setExtendedRuntimePerfKnobConfig__ExtendedRuntimePerfKnobConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ae22a98cd8cc64b1caaab9463f2e3ae44"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setExtendedRuntimePerfKnobConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">extendedRuntimePerfKnobConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig32setExtendedRuntimePerfKnobConfigERK29ExtendedRuntimePerfKnobConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig14setDebugConfigERK11DebugConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig14setDebugConfigERK11DebugConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig14setDebugConfigERK11DebugConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setDebugConfig__DebugConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a2644f7d8f8156fab18febe21da42c6a4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDebugConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">debugConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14setDebugConfigERK11DebugConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig19setRecvPollPeriodMsERK10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig19setRecvPollPeriodMsERK10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig19setRecvPollPeriodMsERK10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setRecvPollPeriodMs__SizeType32CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ae716a2e12980f78e8d001e629a84663f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRecvPollPeriodMs</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">recvPollPeriodMs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig19setRecvPollPeriodMsERK10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setMaxSeqIdleMicrosecondsE8uint64_t">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig25setMaxSeqIdleMicrosecondsE8uint64_t"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig25setMaxSeqIdleMicrosecondsE8uint64_t"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxSeqIdleMicroseconds__uint64_t"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1abe0a4f10c1a9e4864cac511f3ad5320e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxSeqIdleMicroseconds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">uint64_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setMaxSeqIdleMicrosecondsE8uint64_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setSpecDecConfigERK25SpeculativeDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16setSpecDecConfigERK25SpeculativeDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16setSpecDecConfigERK25SpeculativeDecodingConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setSpecDecConfig__SpeculativeDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af7a9647b5d113457510b08ab62f6ab92"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSpecDecConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">specDecConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setSpecDecConfigERK25SpeculativeDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setGuidedDecodingConfigERK20GuidedDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23setGuidedDecodingConfigERK20GuidedDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23setGuidedDecodingConfigERK20GuidedDecodingConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setGuidedDecodingConfig__GuidedDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ad65dc27bdc3214e55abc302239535419"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setGuidedDecodingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">guidedDecodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setGuidedDecodingConfigERK20GuidedDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultMaxSeqIdleMicrosecondsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig30kDefaultMaxSeqIdleMicrosecondsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig30kDefaultMaxSeqIdleMicrosecondsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::kDefaultMaxSeqIdleMicroseconds__uint64_t"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a4cb2fb0a75c587a97ceabfb7556bb4f1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="n"><span class="pre">uint64_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultMaxSeqIdleMicroseconds</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">180000000</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultMaxSeqIdleMicrosecondsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultIterStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig30kDefaultIterStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig30kDefaultIterStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::kDefaultIterStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a85af795ceffac779ed4b9e072905351f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultIterStatsMaxIterations</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1000</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30kDefaultIterStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig33kDefaultRequestStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig33kDefaultRequestStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig33kDefaultRequestStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::kDefaultRequestStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a4f8c59a591914a7df498b1964606bb57"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultRequestStatsMaxIterations</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig33kDefaultRequestStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a94c48a5d5ec190006f387c7a2b6704a8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxBeamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE" title="Link to this definition"></a><br /></dt>
<dd><p>The beam width value of requests that will be sent to the executor. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mSchedulerConfig__SchedulerConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ab1625bee6fca601abb5f1eb6b6355b5b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSchedulerConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The scheduler configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mKvCacheConfig__KvCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a9c032445833352f9d7878d16a145ed4f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mKvCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The KV cache configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mEnableChunkedContext__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a82112a8e221adc965968a1f127ba473e"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableChunkedContext</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE" title="Link to this definition"></a><br /></dt>
<dd><p>The KV cache configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mNormalizeLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a84ac3e0d9136d145a47d24f5eef306c3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNormalizeLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if log probabilities should be normalized or not. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mIterStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a201b184dee2e18455e22f45e645ac113"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mIterStatsMaxIterations</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the maximum number of iterations for which to keep statistics. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mRequestStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a185a12d03febdc22bf56aedb1b05a80b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRequestStatsMaxIterations</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the maximum number of iterations for which to keep per-request statistics. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mBatchingType__BatchingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a59f638d7780b319b27c606352db470d3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBatchingType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The type of batching strategy to use. See BatchingType. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxBatchSize__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a02960bc64feef914f78fc0712796535f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>The max batch size of requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxNumTokens__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a0b08ae45ad7fadece12e0112f0a6e06b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxNumTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The max number of tokens per batch. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mParallelConfig__std::optional:ParallelConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a723466ec6c98eb16b4e8afc69092702d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mParallelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The parallel execution configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mPeftCacheConfig__std::optional:PeftCacheConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aa36980dc3db09507bf58db6d5323294d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPeftCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mLogitsPostProcessorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig26mLogitsPostProcessorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig26mLogitsPostProcessorConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mLogitsPostProcessorConfig__std::optional:LogitsPostProcessorConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a969008c97bd9bdc4d054007a7fc84556"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE" title="tensorrt_llm::executor::LogitsPostProcessorConfig"><span class="n"><span class="pre">LogitsPostProcessorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogitsPostProcessorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mLogitsPostProcessorConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Logits post processor configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mDecodingConfig__std::optional:DecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a0861d3ae711ae600f5f6596128990a66"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Decoding configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mGpuWeightsPercent__float"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a418f075a4e8f30f52c45cbb6cffec2d5"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpuWeightsPercent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE" title="Link to this definition"></a><br /></dt>
<dd><p>GPU weights percent for weight streaming. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxQueueSizeE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxQueueSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxQueueSizeE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxQueueSize__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a90656867544080bda93c0fac2739fafd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxQueueSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxQueueSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>The maximum number of requests allowed in queue before rejecting new requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig30mExtendedRuntimePerfKnobConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig30mExtendedRuntimePerfKnobConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig30mExtendedRuntimePerfKnobConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mExtendedRuntimePerfKnobConfig__ExtendedRuntimePerfKnobConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ab97f65317c339699f4cf2ee795aff3f4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mExtendedRuntimePerfKnobConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig30mExtendedRuntimePerfKnobConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Config for perf knobs that can be set in runtime. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig12mDebugConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig12mDebugConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig12mDebugConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mDebugConfig__std::optional:DebugConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a3b38d197adb1142cd08a9f22c3b20378"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDebugConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig12mDebugConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Debugging configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig17mRecvPollPeriodMsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig17mRecvPollPeriodMsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig17mRecvPollPeriodMsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mRecvPollPeriodMs__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aedf0da0e27422613d74b042a30fdcfe0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRecvPollPeriodMs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17mRecvPollPeriodMsE" title="Link to this definition"></a><br /></dt>
<dd><p>The time in ms between polls for new communication in orchestrator mode. Use 0 for busy loop. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mMaxSeqIdleMicrosecondsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23mMaxSeqIdleMicrosecondsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23mMaxSeqIdleMicrosecondsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxSeqIdleMicroseconds__uint64_t"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a21914eda2bab89712790d945d31d0383"></span><span class="n"><span class="pre">uint64_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxSeqIdleMicroseconds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mMaxSeqIdleMicrosecondsE" title="Link to this definition"></a><br /></dt>
<dd><p>The maximum time in microseconds a scheduled request can remain idle before getting terminated. Default is 3 minutes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mSpeculativeDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig26mSpeculativeDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig26mSpeculativeDecodingConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mSpeculativeDecodingConfig__std::optional:SpeculativeDecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac0e289586d6a768ff070a8c2c16645ad"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSpeculativeDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mSpeculativeDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The speculative decoding configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mGuidedDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig21mGuidedDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig21mGuidedDecodingConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mGuidedDecodingConfig__std::optional:GuidedDecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acfdc8fbac268be88a5df41cd93f68b94"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGuidedDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mGuidedDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The guided decoding configuration. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the runtime perf knobs. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig29ExtendedRuntimePerfKnobConfigEbbb10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig29ExtendedRuntimePerfKnobConfigEbbb10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig29ExtendedRuntimePerfKnobConfigEbbb10SizeType32"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::ExtendedRuntimePerfKnobConfig__b.b.b.SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1ac0c7ab9509bc9ddc4b26d51385170c15"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">multiBlockMode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableContextFMHAFP32Acc</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">cudaGraphMode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">cudaGraphCacheSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig29ExtendedRuntimePerfKnobConfigEbbb10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigeqERK29ExtendedRuntimePerfKnobConfig">
<span id="_CPPv3NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigeqERK29ExtendedRuntimePerfKnobConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigeqERK29ExtendedRuntimePerfKnobConfig"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::eq-operator__ExtendedRuntimePerfKnobConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1ad70b3fe8357913ce8033d452caae3c4f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigeqERK29ExtendedRuntimePerfKnobConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17getMultiBlockModeEv">
<span id="_CPPv3NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17getMultiBlockModeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17getMultiBlockModeEv"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getMultiBlockModeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1ac2d433f7f25ebab808b222785e9959ac"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMultiBlockMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17getMultiBlockModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27getEnableContextFMHAFP32AccEv">
<span id="_CPPv3NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27getEnableContextFMHAFP32AccEv"></span><span id="_CPPv2NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27getEnableContextFMHAFP32AccEv"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getEnableContextFMHAFP32AccC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1aaabc1b1388ee0a1332e2336f38f778ea"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableContextFMHAFP32Acc</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27getEnableContextFMHAFP32AccEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16getCudaGraphModeEv">
<span id="_CPPv3NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16getCudaGraphModeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16getCudaGraphModeEv"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getCudaGraphModeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1a30839851a4c06dc9a3444f46dea2ba59"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCudaGraphMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16getCudaGraphModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21getCudaGraphCacheSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21getCudaGraphCacheSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21getCudaGraphCacheSizeEv"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::getCudaGraphCacheSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1a96c4d3c54bac53bf6d362c575c198a52"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCudaGraphCacheSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21getCudaGraphCacheSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17setMultiBlockModeEb">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17setMultiBlockModeEb"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17setMultiBlockModeEb"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setMultiBlockMode__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1a68d48578b7b2a0dae5ca6e040580ae6e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMultiBlockMode</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">multiBlockMode</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig17setMultiBlockModeEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27setEnableContextFMHAFP32AccEb">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27setEnableContextFMHAFP32AccEb"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27setEnableContextFMHAFP32AccEb"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setEnableContextFMHAFP32Acc__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1af6f642b8d52ac9d7a7ebb308b2f52585"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEnableContextFMHAFP32Acc</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableContextFMHAFP32Acc</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig27setEnableContextFMHAFP32AccEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16setCudaGraphModeEb">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16setCudaGraphModeEb"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16setCudaGraphModeEb"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setCudaGraphMode__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1a0a6a0d8f7f9ee18479096186a67dccb1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCudaGraphMode</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">cudaGraphMode</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig16setCudaGraphModeEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21setCudaGraphCacheSizeE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21setCudaGraphCacheSizeE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21setCudaGraphCacheSizeE10SizeType32"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::setCudaGraphCacheSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1aa9db12e2afad4881208fe5ccadc5ec0f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCudaGraphCacheSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">cacheSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig21setCudaGraphCacheSizeE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig15mMultiBlockModeE">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig15mMultiBlockModeE"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig15mMultiBlockModeE"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mMultiBlockMode__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1a40f5a30741838e416395fba820a111cd"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMultiBlockMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig15mMultiBlockModeE" title="Link to this definition"></a><br /></dt>
<dd><p>Control if multi block mode should be enabled or not. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig25mEnableContextFMHAFP32AccE">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig25mEnableContextFMHAFP32AccE"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig25mEnableContextFMHAFP32AccE"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mEnableContextFMHAFP32Acc__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1abec03437378a48e9fe254b0529c9a2df"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableContextFMHAFP32Acc</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig25mEnableContextFMHAFP32AccE" title="Link to this definition"></a><br /></dt>
<dd><p>If enable FMHA runner FP32 accumulation. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig14mCudaGraphModeE">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig14mCudaGraphModeE"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig14mCudaGraphModeE"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mCudaGraphMode__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1affd41e6596e980c0f37337e91ef7a6b6"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCudaGraphMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig14mCudaGraphModeE" title="Link to this definition"></a><br /></dt>
<dd><p>Control if enable cuda graph. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig19mCudaGraphCacheSizeE">
<span id="_CPPv3N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig19mCudaGraphCacheSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig19mCudaGraphCacheSizeE"></span><span id="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig::mCudaGraphCacheSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExtendedRuntimePerfKnobConfig_1a944b111b52935ea2ce1054704d8c0c23"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCudaGraphCacheSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfig19mCudaGraphCacheSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of cuda graphs to be cached in the runtime. The larger the cache, the better the perf, but more GPU memory is consumed. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfigE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for speculative decoding with external draft tokens. Allows to include draft tokens, draft logits and specify acceptance threshold. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEERKNSt8optionalIbEE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEERKNSt8optionalIbEE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEERKNSt8optionalIbEE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::ExternalDraftTokensConfig__VecTokens.std::optional:Tensor:.std::optional:FloatType:CR.std::optional:b:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a81c077472035062e1e68034d1e762635"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tokens</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">acceptanceThreshold</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">fastLogits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEERKNSt8optionalIbEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1ad99090a218772e934a91558571ea4f67"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getLogitsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a23d59333fbfd34696f1f9ef1e386182d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogits</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getAcceptanceThresholdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a179d3620d160c642b515b48c6ccaea22"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getAcceptanceThreshold</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig13getFastLogitsEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig13getFastLogitsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig13getFastLogitsEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getFastLogitsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1ad0eae155e1cc3957e6edd3226b430ca9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFastLogits</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig13getFastLogitsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mTokens__VecTokens"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a648382f7b513a2fb82330540687833e3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The draft tokens. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mLogits__std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1add7e98a70f09f15c3390a39a7673b5a5"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>The draft logits. Expected shape: [num_draft_tokens, vocab_size]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mAcceptanceThreshold__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1abcc619153ad5a5e2c5bf08940eda9064"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mAcceptanceThreshold</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE" title="Link to this definition"></a><br /></dt>
<dd><p>The acceptance threshold. Must be &gt; 0.f and &lt;= 1.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig11mFastLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig11mFastLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig11mFastLogitsE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mFastLogits__std::optional:b:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a799828aa3a4ecf95517f5f7227cef430"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFastLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig11mFastLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>Use direct transfer for draft logits. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfigE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Guided decoding configurations for executor. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a8a09e91495919291c648a0ef8c53d912"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GuidedDecodingBackend</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackend9kXGRAMMARE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackend9kXGRAMMARE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackend9kXGRAMMARE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a8a09e91495919291c648a0ef8c53d912af9ef3c75182d09e9a4f80fee7ae4d3e6"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kXGRAMMAR</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackend9kXGRAMMARE" title="Link to this definition"></a><br /></dt>
<dd><p>Enable guided decoding with XGrammar backend. </p>
</dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig20GuidedDecodingConfigE21GuidedDecodingBackendNSt8optionalINSt6vectorINSt6stringEEEEENSt8optionalINSt6stringEEENSt8optionalINSt6vectorI11TokenIdTypeEEEE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig20GuidedDecodingConfigE21GuidedDecodingBackendNSt8optionalINSt6vectorINSt6stringEEEEENSt8optionalINSt6stringEEENSt8optionalINSt6vectorI11TokenIdTypeEEEE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig20GuidedDecodingConfigE21GuidedDecodingBackendNSt8optionalINSt6vectorINSt6stringEEEEENSt8optionalINSt6stringEEENSt8optionalINSt6vectorI11TokenIdTypeEEEE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingConfig__GuidedDecodingBackend.std::optional:std::vector:ss::.std::optional:ss:.std::optional:std::vector:TokenIdType::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a9d6fc21aebb2c45bf09d150249f10683"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE" title="tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingBackend"><span class="n"><span class="pre">GuidedDecodingBackend</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">backend</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encodedVocab</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tokenizerStr</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stopTokenIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig20GuidedDecodingConfigE21GuidedDecodingBackendNSt8optionalINSt6vectorINSt6stringEEEEENSt8optionalINSt6stringEEENSt8optionalINSt6vectorI11TokenIdTypeEEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfigeqERK20GuidedDecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingConfigeqERK20GuidedDecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingConfigeqERK20GuidedDecodingConfig"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::eq-operator__GuidedDecodingConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a71e9e9dffeefe7149c0859c82b984ece"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfigeqERK20GuidedDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig10setBackendERK21GuidedDecodingBackend">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig10setBackendERK21GuidedDecodingBackend"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig10setBackendERK21GuidedDecodingBackend"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::setBackend__GuidedDecodingBackendCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1ab79112af809cfce8fa5ea720f8eb2551"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBackend</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE" title="tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingBackend"><span class="n"><span class="pre">GuidedDecodingBackend</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">backend</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig10setBackendERK21GuidedDecodingBackend" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig10getBackendEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingConfig10getBackendEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingConfig10getBackendEv"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::getBackendC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1aacf9c80ebd33edb32b34f354210e9b12"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE" title="tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingBackend"><span class="n"><span class="pre">GuidedDecodingBackend</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBackend</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig10getBackendEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setEncodedVocabERKNSt6vectorINSt6stringEEE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig15setEncodedVocabERKNSt6vectorINSt6stringEEE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig15setEncodedVocabERKNSt6vectorINSt6stringEEE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::setEncodedVocab__std::vector:ss:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1ac33775dbefa1b6f763114d88f8a9cd0b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEncodedVocab</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encodedVocab</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setEncodedVocabERKNSt6vectorINSt6stringEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getEncodedVocabEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingConfig15getEncodedVocabEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingConfig15getEncodedVocabEv"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::getEncodedVocabC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a59c96bec37ce22abbef8f8a3188350fc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEncodedVocab</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getEncodedVocabEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setTokenizerStrERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig15setTokenizerStrERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig15setTokenizerStrERKNSt6stringE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::setTokenizerStr__ssCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1aa0d470301f6e1dfff0f61c8fb45d8c58"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTokenizerStr</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tokenizerStr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setTokenizerStrERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getTokenizerStrEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingConfig15getTokenizerStrEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingConfig15getTokenizerStrEv"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::getTokenizerStrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a126b7221f9d2d73e2eafe6c587439b68"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTokenizerStr</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getTokenizerStrEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setStopTokenIdsERKNSt6vectorI11TokenIdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig15setStopTokenIdsERKNSt6vectorI11TokenIdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig15setStopTokenIdsERKNSt6vectorI11TokenIdTypeEE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::setStopTokenIds__std::vector:TokenIdType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a23a578a4affc585b4bac7c41f15d34e9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setStopTokenIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stopTokenIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig15setStopTokenIdsERKNSt6vectorI11TokenIdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getStopTokenIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingConfig15getStopTokenIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingConfig15getStopTokenIdsEv"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::getStopTokenIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a7dd0f7e137c5f31e34eb1bccf60d188f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getStopTokenIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig15getStopTokenIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig8validateEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingConfig8validateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingConfig8validateEv"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::validateC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1aa069dcb5d299be6720d2ccb3929fc0a1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">validate</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingConfig8validateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig8mBackendE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig8mBackendE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig8mBackendE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::mBackend__GuidedDecodingBackend"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1afc33e7f31234b08feb15f93b858e3c83"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig21GuidedDecodingBackendE" title="tensorrt_llm::executor::GuidedDecodingConfig::GuidedDecodingBackend"><span class="n"><span class="pre">GuidedDecodingBackend</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBackend</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig8mBackendE" title="Link to this definition"></a><br /></dt>
<dd><p>Guided decoding backend. Currently supports XGrammar. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mEncodedVocabE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig13mEncodedVocabE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig13mEncodedVocabE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::mEncodedVocab__std::optional:std::vector:ss::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1af2615dccce382e26f070b3afb89ff3c7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEncodedVocab</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mEncodedVocabE" title="Link to this definition"></a><br /></dt>
<dd><p>Encoded vocabulary. For a huggingface tokenizer, it can be extracted by: </p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">encoded_vocab</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">get_vocab</span><span class="p">()</span>
<span class="n">encoded_vocab</span> <span class="o">=</span> <span class="p">[</span><span class="n">token</span> <span class="k">for</span> <span class="n">token</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">encoded_vocab</span><span class="o">.</span><span class="n">items</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">])]</span>
</pre></div>
</div>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mTokenizerStrE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig13mTokenizerStrE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig13mTokenizerStrE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::mTokenizerStr__std::optional:ss:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1adc4800cf6648cfa71528276372de8197"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTokenizerStr</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mTokenizerStrE" title="Link to this definition"></a><br /></dt>
<dd><p>Tokenizer string. For a huggingface fast tokenizer, it can be extracted by: </p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">tokenizer_str</span> <span class="o">=</span> <span class="n">tokenizer</span><span class="o">.</span><span class="n">backend_tokenizer</span><span class="o">.</span><span class="n">to_str</span><span class="p">()</span>
</pre></div>
</div>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mStopTokenIdsE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingConfig13mStopTokenIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingConfig13mStopTokenIdsE"></span><span id="tensorrt_llm::executor::GuidedDecodingConfig::mStopTokenIds__std::optional:std::vector:TokenIdType::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingConfig_1a405c8a802aadfaa66102519c00c9be33"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mStopTokenIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfig13mStopTokenIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stop token ids. If not provided, it can be automatically detected. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParamsE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParamsE"></span><span id="tensorrt_llm::executor::GuidedDecodingParams"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GuidedDecodingParams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Guided decoding parameters for a request. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a3e79896af4d977fc8eedb6dae21c5946"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GuideType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType5kJSONE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams9GuideType5kJSONE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams9GuideType5kJSONE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a3e79896af4d977fc8eedb6dae21c5946a7820f24be07e92b4ed8ef25f129cf422"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kJSON</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType5kJSONE" title="Link to this definition"></a><br /></dt>
<dd><p>The generated text is amenable to json format. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType12kJSON_SCHEMAE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams9GuideType12kJSON_SCHEMAE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams9GuideType12kJSON_SCHEMAE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a3e79896af4d977fc8eedb6dae21c5946a08c4ca8b86b0f06177c4d0b7041e938f"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kJSON_SCHEMA</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType12kJSON_SCHEMAE" title="Link to this definition"></a><br /></dt>
<dd><p>The generated text is amenable to json format with additional user-specified restrictions, namely schema. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType6kREGEXE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams9GuideType6kREGEXE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams9GuideType6kREGEXE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a3e79896af4d977fc8eedb6dae21c5946ae6485c71429ab44931687b52af697e4f"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kREGEX</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType6kREGEXE" title="Link to this definition"></a><br /></dt>
<dd><p>The generated text is amenable to the user-specified regular expression. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType13kEBNF_GRAMMARE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams9GuideType13kEBNF_GRAMMARE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams9GuideType13kEBNF_GRAMMARE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a3e79896af4d977fc8eedb6dae21c5946ae71f648967026c8d1d393580be763498"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kEBNF_GRAMMAR</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideType13kEBNF_GRAMMARE" title="Link to this definition"></a><br /></dt>
<dd><p>The generated text is amenable to the user-specified extended Backus-Naur form (EBNF) grammar. EBNF grammar is widely-used to express context-free grammars. </p>
</dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams20GuidedDecodingParamsE9GuideTypeNSt8optionalINSt6stringEEE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams20GuidedDecodingParamsE9GuideTypeNSt8optionalINSt6stringEEE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams20GuidedDecodingParamsE9GuideTypeNSt8optionalINSt6stringEEE"></span><span id="tensorrt_llm::executor::GuidedDecodingParams::GuidedDecodingParams__GuideType.std::optional:ss:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a6ca5a46030ce345f2d0c61430af3268d"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">GuidedDecodingParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE" title="tensorrt_llm::executor::GuidedDecodingParams::GuideType"><span class="n"><span class="pre">GuideType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">guideType</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">guide</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams20GuidedDecodingParamsE9GuideTypeNSt8optionalINSt6stringEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParamseqERK20GuidedDecodingParams">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingParamseqERK20GuidedDecodingParams"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingParamseqERK20GuidedDecodingParams"></span><span id="tensorrt_llm::executor::GuidedDecodingParams::eq-operator__GuidedDecodingParamsCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a9895d4a2338b7200a8be03babbd61eaf"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParamseqERK20GuidedDecodingParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParams12getGuideTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingParams12getGuideTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingParams12getGuideTypeEv"></span><span id="tensorrt_llm::executor::GuidedDecodingParams::getGuideTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a11a3c820302df3f1c548a07a040a5c59"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE" title="tensorrt_llm::executor::GuidedDecodingParams::GuideType"><span class="n"><span class="pre">GuideType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGuideType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParams12getGuideTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParams8getGuideEv">
<span id="_CPPv3NK12tensorrt_llm8executor20GuidedDecodingParams8getGuideEv"></span><span id="_CPPv2NK12tensorrt_llm8executor20GuidedDecodingParams8getGuideEv"></span><span id="tensorrt_llm::executor::GuidedDecodingParams::getGuideC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a86814394d79a4e7550e3faa7d040bb74"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGuide</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor20GuidedDecodingParams8getGuideEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams10mGuideTypeE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams10mGuideTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams10mGuideTypeE"></span><span id="tensorrt_llm::executor::GuidedDecodingParams::mGuideType__GuideType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1ac2f74b2d1059b890383b2225bc0a7c53"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams9GuideTypeE" title="tensorrt_llm::executor::GuidedDecodingParams::GuideType"><span class="n"><span class="pre">GuideType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGuideType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams10mGuideTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The guide type. See GuideType. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams6mGuideE">
<span id="_CPPv3N12tensorrt_llm8executor20GuidedDecodingParams6mGuideE"></span><span id="_CPPv2N12tensorrt_llm8executor20GuidedDecodingParams6mGuideE"></span><span id="tensorrt_llm::executor::GuidedDecodingParams::mGuide__std::optional:ss:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1GuidedDecodingParams_1a80cc2ed848932eed934f79a1f5cdcc8e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGuide</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParams6mGuideE" title="Link to this definition"></a><br /></dt>
<dd><p>The detailed guide string. It could be a json schema, a regular expression or a EBNF grammar depending on mGuideType. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerializationE">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerializationE"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerializationE"></span><span id="tensorrt_llm::executor::JsonSerialization"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">JsonSerialization</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerializationE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Class with utility functions to serialize statistics to json string. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats"></span><span id="tensorrt_llm::executor::JsonSerialization::toJsonStr__IterationStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization_1abb7c3366de8460accfc20764a1c820c6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toJsonStr</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterationStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to convert an iterationStats struct to a json serialized string. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration"></span><span id="tensorrt_llm::executor::JsonSerialization::toJsonStr__RequestStatsPerIterationCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization_1a8734846c416a9d9199d72c0fc6429374"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toJsonStr</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStatsPerIter</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to convert a requestStatsPerIteration struct to a json serialized string. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats"></span><span id="tensorrt_llm::executor::JsonSerialization::toJsonStr__RequestStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization_1a7cbb6808bdbace0121a04044ec044854"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toJsonStr</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to convert a requestStats struct to a json serialized string. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfigE"></span><span id="tensorrt_llm::executor::KvCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the KV cache. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEbRKNSt8optionalI9FloatTypeEENSt8optionalI17RetentionPriorityEE6size_tRKNSt8optionalIN12tensorrt_llm7runtime15RuntimeDefaultsEEE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEbRKNSt8optionalI9FloatTypeEENSt8optionalI17RetentionPriorityEE6size_tRKNSt8optionalIN12tensorrt_llm7runtime15RuntimeDefaultsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEbRKNSt8optionalI9FloatTypeEENSt8optionalI17RetentionPriorityEE6size_tRKNSt8optionalIN12tensorrt_llm7runtime15RuntimeDefaultsEEE"></span><span id="tensorrt_llm::executor::KvCacheConfig::KvCacheConfig__b.std::optional:SizeType32:CR.std::optional:std::vector:SizeType32::CR.std::optional:SizeType32:CR.std::optional:FloatType:CR.std::optional:s:CR.b.std::optional:FloatType:CR.std::optional:RetentionPriority:.s.std::optional:tensorrt_llm::runtime::RuntimeDefaults:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a648ee4acd774abcac5de49495b18ec33"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableBlockReuse</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">maxTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">maxAttentionWindowVec</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sinkTokenLength</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">freeGpuMemoryFraction</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">hostCacheSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">onboardBlocks</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">crossKvCacheFraction</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">secondaryOffloadMinPriority</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">eventBufferMaxSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime15RuntimeDefaultsE" title="tensorrt_llm::runtime::RuntimeDefaults"><span class="n"><span class="pre">RuntimeDefaults</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">runtimeDefaults</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEbRKNSt8optionalI9FloatTypeEENSt8optionalI17RetentionPriorityEE6size_tRKNSt8optionalIN12tensorrt_llm7runtime15RuntimeDefaultsEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getEnableBlockReuseC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a215347e4303538281e8918d711bee0fe"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableBlockReuse</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getMaxTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a8f8a9fc6161d9f540cdd41e568e6703d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getMaxAttentionWindowVecEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig24getMaxAttentionWindowVecEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig24getMaxAttentionWindowVecEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getMaxAttentionWindowVecC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ab8f28550f538ac770340947ebb99a0aa"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxAttentionWindowVec</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getMaxAttentionWindowVecEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getSinkTokenLengthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ad63d1e7cc46b30f38f2f7811419f2de0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSinkTokenLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getFreeGpuMemoryFractionC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ac216ec0143b80ed2538d54e27ad632d7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFreeGpuMemoryFraction</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig23getCrossKvCacheFractionEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig23getCrossKvCacheFractionEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig23getCrossKvCacheFractionEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getCrossKvCacheFractionC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a57844d3da7f5bfdf467a7f76b5c88278"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCrossKvCacheFraction</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig23getCrossKvCacheFractionEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getHostCacheSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a12ce86b6f1a03e50d65462f94a6f90df"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getHostCacheSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getOnboardBlocksC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a596dc5ef42e0aaeffa728705972635f5"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOnboardBlocks</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig30getSecondaryOffloadMinPriorityEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig30getSecondaryOffloadMinPriorityEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig30getSecondaryOffloadMinPriorityEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getSecondaryOffloadMinPriorityC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1adc7ee22b315728061c46b3936a1bc790"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSecondaryOffloadMinPriority</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig30getSecondaryOffloadMinPriorityEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig21getEventBufferMaxSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig21getEventBufferMaxSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig21getEventBufferMaxSizeEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getEventBufferMaxSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1afff662f63bfe3292d951951884aee2f2"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEventBufferMaxSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig21getEventBufferMaxSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb"></span><span id="tensorrt_llm::executor::KvCacheConfig::setEnableBlockReuse__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a934f7362c2d931489c7df5c86a5c19c1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEnableBlockReuse</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableBlockReuse</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32"></span><span id="tensorrt_llm::executor::KvCacheConfig::setMaxTokens__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1af2998d1bb93ae001bf468e8c6c78214c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setMaxAttentionWindowVecENSt6vectorI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig24setMaxAttentionWindowVecENSt6vectorI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig24setMaxAttentionWindowVecENSt6vectorI10SizeType32EE"></span><span id="tensorrt_llm::executor::KvCacheConfig::setMaxAttentionWindowVec__std::vector:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ad7acf1dcb06442e734a36bce18154387"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxAttentionWindowVec</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxAttentionWindowVec</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setMaxAttentionWindowVecENSt6vectorI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32"></span><span id="tensorrt_llm::executor::KvCacheConfig::setSinkTokenLength__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1abc33d9483367d58b2744884618840a37"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSinkTokenLength</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">sinkTokenLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType"></span><span id="tensorrt_llm::executor::KvCacheConfig::setFreeGpuMemoryFraction__FloatType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1acf2e6b9f0c81b3c1a299e96a6737b587"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setFreeGpuMemoryFraction</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">freeGpuMemoryFraction</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig23setCrossKvCacheFractionE9FloatType">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig23setCrossKvCacheFractionE9FloatType"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig23setCrossKvCacheFractionE9FloatType"></span><span id="tensorrt_llm::executor::KvCacheConfig::setCrossKvCacheFraction__FloatType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a2985e832738877a2d522ded1d630a7ed"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCrossKvCacheFraction</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">crossKvCacheFraction</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig23setCrossKvCacheFractionE9FloatType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t"></span><span id="tensorrt_llm::executor::KvCacheConfig::setHostCacheSize__s"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ab23edc123b320ef59c4c4117e7c97117"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setHostCacheSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">hostCacheSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb"></span><span id="tensorrt_llm::executor::KvCacheConfig::setOnboardBlocks__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ac1ff38e4da7ddf084bc4ad5192257193"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOnboardBlocks</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">onboardBlocks</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig30setSecondaryOffloadMinPriorityENSt8optionalI17RetentionPriorityEE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig30setSecondaryOffloadMinPriorityENSt8optionalI17RetentionPriorityEE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig30setSecondaryOffloadMinPriorityENSt8optionalI17RetentionPriorityEE"></span><span id="tensorrt_llm::executor::KvCacheConfig::setSecondaryOffloadMinPriority__std::optional:RetentionPriority:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ae4dcc95f8c89af99242d43b806dd189d"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSecondaryOffloadMinPriority</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">secondaryOffloadMinPriority</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig30setSecondaryOffloadMinPriorityENSt8optionalI17RetentionPriorityEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig21setEventBufferMaxSizeE6size_t">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig21setEventBufferMaxSizeE6size_t"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig21setEventBufferMaxSizeE6size_t"></span><span id="tensorrt_llm::executor::KvCacheConfig::setEventBufferMaxSize__s"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1acaa651dc98ad9d1ed7ce153315bcea93"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEventBufferMaxSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">eventBufferMaxSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig21setEventBufferMaxSizeE6size_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig34fillEmptyFieldsFromRuntimeDefaultsEN12tensorrt_llm7runtime15RuntimeDefaultsE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig34fillEmptyFieldsFromRuntimeDefaultsEN12tensorrt_llm7runtime15RuntimeDefaultsE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig34fillEmptyFieldsFromRuntimeDefaultsEN12tensorrt_llm7runtime15RuntimeDefaultsE"></span><span id="tensorrt_llm::executor::KvCacheConfig::fillEmptyFieldsFromRuntimeDefaults__tensorrt_llm::runtime::RuntimeDefaults"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1afbaa7fe2589b066f9e95d85fbae894c9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">fillEmptyFieldsFromRuntimeDefaults</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime15RuntimeDefaultsE" title="tensorrt_llm::runtime::RuntimeDefaults"><span class="n"><span class="pre">RuntimeDefaults</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">runtimeDefaults</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig34fillEmptyFieldsFromRuntimeDefaultsEN12tensorrt_llm7runtime15RuntimeDefaultsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mEnableBlockReuse__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a60551af15587436a83fb1ca17a82878f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableBlockReuse</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if KV cache blocks can be reused for different requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mMaxTokens__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a06bf9092bf1b1abedc4c256edb1be59a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The maximum number of tokens that should be stored in the KV cache If both mMaxTokens and mFreeGpuMemoryFraction are specified, memory corresponding to the minimum will be allocated. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mMaxAttentionWindowVecE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig22mMaxAttentionWindowVecE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig22mMaxAttentionWindowVecE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mMaxAttentionWindowVec__std::optional:std::vector:SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1aaf941396e46ebfc43d3f8908845b8d1e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxAttentionWindowVec</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mMaxAttentionWindowVecE" title="Link to this definition"></a><br /></dt>
<dd><p>Size of the attention window for each sequence. Only the last mMaxAttentionWindow tokens of each sequence will be stored in the KV cache. Different layers may have different max attention window sizes. If the number of elements in mMaxAttentionWindowVec is less than the number of layers, mMaxAttentionWindowVec will be repeated multiple times to the number of layers. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mSinkTokenLength__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a84f967e8a537c1060870714612acc0e4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSinkTokenLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of sink tokens (tokens to always keep in attention window) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mFreeGpuMemoryFraction__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ac48a05dd2ed2669970f12495e49c3cd1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFreeGpuMemoryFraction</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE" title="Link to this definition"></a><br /></dt>
<dd><p>The fraction of GPU memory fraction that should be allocated for the KV cache. Default is 90%. If both mMaxTokens and mFreeGpuMemoryFraction are specified, memory corresponding to the minimum will be allocated. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig21mCrossKvCacheFractionE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig21mCrossKvCacheFractionE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig21mCrossKvCacheFractionE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mCrossKvCacheFraction__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a331f2b5c3a2b92646a851936c1386c66"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCrossKvCacheFraction</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig21mCrossKvCacheFractionE" title="Link to this definition"></a><br /></dt>
<dd><p>The fraction of the KV Cache memory should be reserved for cross attention If set to p, self attention will use 1-p of KV Cache memory and cross attention will use p of KV Cache memory. Default is 50%. Should only be set when using encoder-decoder model. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mHostCacheSize__std::optional:s:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a3ec456502080d1b581b3619ce46a0ba6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mHostCacheSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>Size of secondary memory pool in bytes. Default is 0. Having a secondary memory pool increases KV cache block reuse potential. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mOnboardBlocks__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ad16b0f0939192c1b97e6d2e905c3a1cc"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOnboardBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls whether offloaded blocks should be onboarded back into primary memory before being reused. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig28mSecondaryOffloadMinPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig28mSecondaryOffloadMinPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig28mSecondaryOffloadMinPriorityE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mSecondaryOffloadMinPriority__std::optional:RetentionPriority:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a7aae6ab2ac9a6e1c677e88898f366a15"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSecondaryOffloadMinPriority</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig28mSecondaryOffloadMinPriorityE" title="Link to this definition"></a><br /></dt>
<dd><p>Only blocks with priority &gt; mSecondaryOfflineMinPriority can be offloaded to secondary memory. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig19mEventBufferMaxSizeE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig19mEventBufferMaxSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig19mEventBufferMaxSizeE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mEventBufferMaxSize__s"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a6ef7d949ae48a5f5862c995d70b5f84f"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEventBufferMaxSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig19mEventBufferMaxSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>Max size of the KV cache event buffer. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheCreatedDataE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheCreatedDataE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheCreatedDataE"></span><span id="tensorrt_llm::executor::KVCacheCreatedData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheCreatedData"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheCreatedData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheCreatedDataE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheCreatedData22numBlocksPerCacheLevelE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheCreatedData22numBlocksPerCacheLevelE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheCreatedData22numBlocksPerCacheLevelE"></span><span id="tensorrt_llm::executor::KVCacheCreatedData::numBlocksPerCacheLevel__std::vector:SizeType32:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheCreatedData_1abae9e965629624bc13a99b0b4b72dcf9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numBlocksPerCacheLevel</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheCreatedData22numBlocksPerCacheLevelE" title="Link to this definition"></a><br /></dt>
<dd><p>The amount of blocks at each cache level. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KVCacheEventE">
<span id="_CPPv3N12tensorrt_llm8executor12KVCacheEventE"></span><span id="_CPPv2N12tensorrt_llm8executor12KVCacheEventE"></span><span id="tensorrt_llm::executor::KVCacheEvent"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEvent"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheEvent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEventE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KVCacheEvent12KVCacheEventE6IdType16KVCacheEventData">
<span id="_CPPv3N12tensorrt_llm8executor12KVCacheEvent12KVCacheEventE6IdType16KVCacheEventData"></span><span id="_CPPv2N12tensorrt_llm8executor12KVCacheEvent12KVCacheEventE6IdType16KVCacheEventData"></span><span id="tensorrt_llm::executor::KVCacheEvent::KVCacheEvent__IdType.KVCacheEventData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEvent_1a9fcd54bacda6f5c09704b9aaad464d26"></span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheEvent</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">eventId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDataE" title="tensorrt_llm::executor::KVCacheEventData"><span class="n"><span class="pre">KVCacheEventData</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">data</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEvent12KVCacheEventE6IdType16KVCacheEventData" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KVCacheEvent7eventIdE">
<span id="_CPPv3N12tensorrt_llm8executor12KVCacheEvent7eventIdE"></span><span id="_CPPv2N12tensorrt_llm8executor12KVCacheEvent7eventIdE"></span><span id="tensorrt_llm::executor::KVCacheEvent::eventId__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEvent_1ac35ffc03619ba4c47d466b1c64a9e477"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">eventId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEvent7eventIdE" title="Link to this definition"></a><br /></dt>
<dd><p>The unique id of this event. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KVCacheEvent4dataE">
<span id="_CPPv3N12tensorrt_llm8executor12KVCacheEvent4dataE"></span><span id="_CPPv2N12tensorrt_llm8executor12KVCacheEvent4dataE"></span><span id="tensorrt_llm::executor::KVCacheEvent::data__KVCacheEventData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEvent_1a89f7a015393de6fd4b3617667debedcc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDataE" title="tensorrt_llm::executor::KVCacheEventData"><span class="n"><span class="pre">KVCacheEventData</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">data</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEvent4dataE" title="Link to this definition"></a><br /></dt>
<dd><p>The data corresponding to this event. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE">
<span id="_CPPv3I0EN12tensorrt_llm8executor16KVCacheEventDiffE"></span><span id="_CPPv2I0EN12tensorrt_llm8executor16KVCacheEventDiffE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEventDiff"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheEventDiff</span></span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor16KVCacheEventDiff8oldValueE">
<span id="_CPPv3N12tensorrt_llm8executor16KVCacheEventDiff8oldValueE"></span><span id="_CPPv2N12tensorrt_llm8executor16KVCacheEventDiff8oldValueE"></span><span id="tensorrt_llm::executor::KVCacheEventDiff::oldValue__T"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEventDiff_1a5eace570b46a93e06db3df99479eece4"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE" title="tensorrt_llm::executor::KVCacheEventDiff::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">oldValue</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDiff8oldValueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor16KVCacheEventDiff8newValueE">
<span id="_CPPv3N12tensorrt_llm8executor16KVCacheEventDiff8newValueE"></span><span id="_CPPv2N12tensorrt_llm8executor16KVCacheEventDiff8newValueE"></span><span id="tensorrt_llm::executor::KVCacheEventDiff::newValue__T"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheEventDiff_1aaf2dcad5ce01791fc1ef4a38ef683ed8"></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE" title="tensorrt_llm::executor::KVCacheEventDiff::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newValue</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor16KVCacheEventDiff8newValueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19KVCacheEventManagerE">
<span id="_CPPv3N12tensorrt_llm8executor19KVCacheEventManagerE"></span><span id="_CPPv2N12tensorrt_llm8executor19KVCacheEventManagerE"></span><span id="tensorrt_llm::executor::KVCacheEventManager"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KVCacheEventManager"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheEventManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManagerE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Exposes a limited set of KV cache manager functionalities. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19KVCacheEventManager19KVCacheEventManagerENSt10shared_ptrIN12tensorrt_llm13batch_manager16kv_cache_manager18BaseKVCacheManagerEEE">
<span id="_CPPv3N12tensorrt_llm8executor19KVCacheEventManager19KVCacheEventManagerENSt10shared_ptrIN12tensorrt_llm13batch_manager16kv_cache_manager18BaseKVCacheManagerEEE"></span><span id="_CPPv2N12tensorrt_llm8executor19KVCacheEventManager19KVCacheEventManagerENSt10shared_ptrIN12tensorrt_llm13batch_manager16kv_cache_manager18BaseKVCacheManagerEEE"></span><span id="tensorrt_llm::executor::KVCacheEventManager::KVCacheEventManager__std::shared_ptr:tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KVCacheEventManager_1acddf2441f42e1f19c7a06ac42b4dac6b"></span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheEventManager</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="tensorrt_llm::batch_manager"><span class="n"><span class="pre">batch_manager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="tensorrt_llm::batch_manager::kv_cache_manager"><span class="n"><span class="pre">kv_cache_manager</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">BaseKVCacheManager</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">kvCacheManager</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManager19KVCacheEventManagerENSt10shared_ptrIN12tensorrt_llm13batch_manager16kv_cache_manager18BaseKVCacheManagerEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19KVCacheEventManager15getLatestEventsENSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor19KVCacheEventManager15getLatestEventsENSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor19KVCacheEventManager15getLatestEventsENSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::KVCacheEventManager::getLatestEvents__std::optional:std::chrono::milliseconds:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KVCacheEventManager_1a42654b341294b34003bea8ed42587946"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">deque</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KVCacheEventE" title="tensorrt_llm::executor::KVCacheEvent"><span class="n"><span class="pre">KVCacheEvent</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLatestEvents</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManager15getLatestEventsENSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Get the latest KV Cache events. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>timeout</strong> – The maximum time to wait for new events. If nullopt, will only return when new events are available, or when the executor instance has shutdown. </p>
</dd>
</dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19KVCacheEventManager14kvCacheManagerE">
<span id="_CPPv3N12tensorrt_llm8executor19KVCacheEventManager14kvCacheManagerE"></span><span id="_CPPv2N12tensorrt_llm8executor19KVCacheEventManager14kvCacheManagerE"></span><span id="tensorrt_llm::executor::KVCacheEventManager::kvCacheManager__std::shared_ptr:tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KVCacheEventManager_1a0a63a4b88e9a375232403da0ed20c403"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_managerE" title="tensorrt_llm::batch_manager"><span class="n"><span class="pre">batch_manager</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm13batch_manager16kv_cache_managerE" title="tensorrt_llm::batch_manager::kv_cache_manager"><span class="n"><span class="pre">kv_cache_manager</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">BaseKVCacheManager</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheManager</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19KVCacheEventManager14kvCacheManagerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheRemovedDataE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheRemovedDataE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheRemovedDataE"></span><span id="tensorrt_llm::executor::KVCacheRemovedData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheRemovedData"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheRemovedData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheRemovedDataE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheRemovedData11blockHashesE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheRemovedData11blockHashesE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheRemovedData11blockHashesE"></span><span id="tensorrt_llm::executor::KVCacheRemovedData::blockHashes__std::vector:IdType:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheRemovedData_1a33b74e147613ec8f12a3fcc441eb6df4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">blockHashes</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheRemovedData11blockHashesE" title="Link to this definition"></a><br /></dt>
<dd><p>The hashes of blocks being removed. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfigE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for the request’s retention in the KV Cache. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigEv">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigEv"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigEv"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::KvCacheRetentionConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1a1a64c8305f3a0ff4d2f7f981baa06c00"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigERKNSt6vectorI25TokenRangeRetentionConfigEE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigERKNSt6vectorI25TokenRangeRetentionConfigEE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigERKNSt6vectorI25TokenRangeRetentionConfigEE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::KvCacheRetentionConfig__std::vector:TokenRangeRetentionConfig:CR.RetentionPriority.std::optional:std::chrono::milliseconds:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1a169602b126b3210e4e48031c319d2a10"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tokenRangeRetentionPriorities</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">decodeRetentionPriority</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE" title="tensorrt_llm::executor::KvCacheRetentionConfig::kDefaultRetentionPriority"><span class="n"><span class="pre">kDefaultRetentionPriority</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decodeDurationMs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig22KvCacheRetentionConfigERKNSt6vectorI25TokenRangeRetentionConfigEE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig29getTokenRangeRetentionConfigsEv">
<span id="_CPPv3NK12tensorrt_llm8executor22KvCacheRetentionConfig29getTokenRangeRetentionConfigsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor22KvCacheRetentionConfig29getTokenRangeRetentionConfigsEv"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::getTokenRangeRetentionConfigsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1a4f7eb9e150abff8c30521b3d2b11f68a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTokenRangeRetentionConfigs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig29getTokenRangeRetentionConfigsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig26getDecodeRetentionPriorityEv">
<span id="_CPPv3NK12tensorrt_llm8executor22KvCacheRetentionConfig26getDecodeRetentionPriorityEv"></span><span id="_CPPv2NK12tensorrt_llm8executor22KvCacheRetentionConfig26getDecodeRetentionPriorityEv"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::getDecodeRetentionPriorityC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1a1be6f95e4aa00372070790c06a606f93"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDecodeRetentionPriority</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig26getDecodeRetentionPriorityEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig19getDecodeDurationMsEv">
<span id="_CPPv3NK12tensorrt_llm8executor22KvCacheRetentionConfig19getDecodeDurationMsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor22KvCacheRetentionConfig19getDecodeDurationMsEv"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::getDecodeDurationMsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1aee9c4b1af0e4f7650565ee3074b5f397"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDecodeDurationMs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig19getDecodeDurationMsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig36getPerBlockRetentionPriorityDurationE10SizeType3210SizeType32">
<span id="_CPPv3NK12tensorrt_llm8executor22KvCacheRetentionConfig36getPerBlockRetentionPriorityDurationE10SizeType3210SizeType32"></span><span id="_CPPv2NK12tensorrt_llm8executor22KvCacheRetentionConfig36getPerBlockRetentionPriorityDurationE10SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::getPerBlockRetentionPriorityDuration__SizeType32.SizeType32C"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1afd4e5ccd65e62f5d6bdba3bef0b116f0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDurationE" title="tensorrt_llm::executor::RetentionPriorityAndDuration"><span class="n"><span class="pre">RetentionPriorityAndDuration</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPerBlockRetentionPriorityDuration</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">blockSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">seqLen</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig36getPerBlockRetentionPriorityDurationE10SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd><p>Convert the token range data into an entry per kv block. Returns a tuple of vectors corresponding to the priorities and durations for each block. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig21kMinRetentionPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig21kMinRetentionPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig21kMinRetentionPriorityE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::kMinRetentionPriority__RetentionPriority"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1ad1775f789f5c8f01dc1546b65bc15750"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMinRetentionPriority</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig21kMinRetentionPriorityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig21kMaxRetentionPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig21kMaxRetentionPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig21kMaxRetentionPriorityE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::kMaxRetentionPriority__RetentionPriority"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1aa5bd07c90434c99284fec59df5590cb6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMaxRetentionPriority</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">100</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig21kMaxRetentionPriorityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::kDefaultRetentionPriority__RetentionPriority"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1a6f575882c968d713624a82ff748b9da8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultRetentionPriority</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">35</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig27mTokenRangeRetentionConfigsE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig27mTokenRangeRetentionConfigsE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig27mTokenRangeRetentionConfigsE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::mTokenRangeRetentionConfigs__std::vector:TokenRangeRetentionConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1aaae6311e465a6fad29fd4cd7d1cd33f0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTokenRangeRetentionConfigs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig27mTokenRangeRetentionConfigsE" title="Link to this definition"></a><br /></dt>
<dd><p>The token ranges and priority levels to update. Ranges must be non-overlapping. For example [(0, 64), (100, 128), (70, 80)] is valid, whereas [(0, 64), (60, 128)] is not. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig24mDecodeRetentionPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig24mDecodeRetentionPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig24mDecodeRetentionPriorityE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::mDecodeRetentionPriority__RetentionPriority"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1a6b1d57ec2bae317c057dde80f5b9cfdd"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodeRetentionPriority</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig24mDecodeRetentionPriorityE" title="Link to this definition"></a><br /></dt>
<dd><p>The priority level to assign to blocks allocated in the decode phase. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig17mDecodeDurationMsE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig17mDecodeDurationMsE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig17mDecodeDurationMsE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::mDecodeDurationMs__std::optional:std::chrono::milliseconds:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1aeff5b4895c608067ea663350021094fe"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodeDurationMs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig17mDecodeDurationMsE" title="Link to this definition"></a><br /></dt>
<dd><p>The duration in ms that decode blocks should remain at their assigned priority level. </p>
</dd></dl>
</div>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>A single entry to set block priorities over a token range. Earlier ranges always take priority over later ones. For example, with a block size of 16, a range of [0, 17] would be applied to the first two blocks. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig25TokenRangeRetentionConfigE10SizeType32NSt8optionalI10SizeType32EE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig25TokenRangeRetentionConfigE10SizeType32NSt8optionalI10SizeType32EE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig25TokenRangeRetentionConfigE10SizeType32NSt8optionalI10SizeType32EE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig::TokenRangeRetentionConfig__SizeType32.std::optional:SizeType32:.RetentionPriority.std::optional:std::chrono::milliseconds:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig_1a01aa276ddfd1389b808557c6d9be6f73"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tokenStart</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tokenEnd</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">priority</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25kDefaultRetentionPriorityE" title="tensorrt_llm::executor::KvCacheRetentionConfig::kDefaultRetentionPriority"><span class="n"><span class="pre">kDefaultRetentionPriority</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">durationMs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig25TokenRangeRetentionConfigE10SizeType32NSt8optionalI10SizeType32EE17RetentionPriorityNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigeqERK25TokenRangeRetentionConfig">
<span id="_CPPv3NK12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigeqERK25TokenRangeRetentionConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigeqERK25TokenRangeRetentionConfig"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig::eq-operator__TokenRangeRetentionConfigCRC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig_1a58c0f14a585320a450319a21b8037824"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigeqERK25TokenRangeRetentionConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10tokenStartE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10tokenStartE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10tokenStartE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig::tokenStart__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig_1abdb050fcdd04f64406f1d72491fdeb1a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tokenStart</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10tokenStartE" title="Link to this definition"></a><br /></dt>
<dd><p>The first token of this range. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8tokenEndE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8tokenEndE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8tokenEndE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig::tokenEnd__std::optional:SizeType32:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig_1a1665679cac7313b073d4906729a255c3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tokenEnd</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8tokenEndE" title="Link to this definition"></a><br /></dt>
<dd><p>The final token of this range. The end is not included in the range. This can be set to std::nullopt to extend the range to the end of the sequence. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8priorityE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8priorityE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8priorityE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig::priority__RetentionPriority"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig_1a72224537a247f5003076f86892833696"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">priority</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig8priorityE" title="Link to this definition"></a><br /></dt>
<dd><p>The priority of this token range. Higher priorities are less likely to be evicted or offloaded. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10durationMsE">
<span id="_CPPv3N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10durationMsE"></span><span id="_CPPv2N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10durationMsE"></span><span id="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig::durationMs__std::optional:std::chrono::milliseconds:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheRetentionConfig_1_1TokenRangeRetentionConfig_1af8953ca7acc4db009d738bc10f70c2ab"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">durationMs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfig10durationMsE" title="Link to this definition"></a><br /></dt>
<dd><p>The duration in ms that the block should remain at the given priority level. Set to std::nullopt to have no expiration time, and keep the block at the given priority level until it gets reclaimed. After the duration has passed, the block will be moved back to the <code class="docutils literal notranslate"><span class="pre">kDefaultRetentionPriority</span></code> level. </p>
</dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockDataE">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockDataE"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockDataE"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheStoredBlockData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockDataE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>An entry for a single block stored into the tree. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData22KVCacheStoredBlockDataE6IdTypeN12tensorrt_llm7runtime15VecUniqueTokensEN12tensorrt_llm7runtime14LoraTaskIdTypeE10SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockData22KVCacheStoredBlockDataE6IdTypeN12tensorrt_llm7runtime15VecUniqueTokensEN12tensorrt_llm7runtime14LoraTaskIdTypeE10SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockData22KVCacheStoredBlockDataE6IdTypeN12tensorrt_llm7runtime15VecUniqueTokensEN12tensorrt_llm7runtime14LoraTaskIdTypeE10SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData::KVCacheStoredBlockData__IdType.tensorrt_llm::runtime::VecUniqueTokens.tensorrt_llm::runtime::LoraTaskIdType.SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData_1a3b7f19c119c1151a610a15482ca57b38"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheStoredBlockData</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">blockHash</span></span>, <a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime15VecUniqueTokensE" title="tensorrt_llm::runtime::VecUniqueTokens"><span class="n"><span class="pre">VecUniqueTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tokens</span></span>, <a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime14LoraTaskIdTypeE" title="tensorrt_llm::runtime::LoraTaskIdType"><span class="n"><span class="pre">LoraTaskIdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">loraId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">cacheLevel</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">priority</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData22KVCacheStoredBlockDataE6IdTypeN12tensorrt_llm7runtime15VecUniqueTokensEN12tensorrt_llm7runtime14LoraTaskIdTypeE10SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData9blockHashE">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockData9blockHashE"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockData9blockHashE"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData::blockHash__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData_1a356cdfeb2a70ae9bd8d4be5c38fc1044"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">blockHash</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData9blockHashE" title="Link to this definition"></a><br /></dt>
<dd><p>The hash of the block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData6tokensE">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockData6tokensE"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockData6tokensE"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData::tokens__tensorrt_llm::runtime::VecUniqueTokens"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData_1a72b14db5340f183fae50810461400907"></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime15VecUniqueTokensE" title="tensorrt_llm::runtime::VecUniqueTokens"><span class="n"><span class="pre">VecUniqueTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData6tokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The unique tokens of the block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData6loraIdE">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockData6loraIdE"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockData6loraIdE"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData::loraId__tensorrt_llm::runtime::LoraTaskIdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData_1aa13325389143a28d6aeafa19b51508c8"></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime14LoraTaskIdTypeE" title="tensorrt_llm::runtime::LoraTaskIdType"><span class="n"><span class="pre">LoraTaskIdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">loraId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData6loraIdE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora task id of the block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData10cacheLevelE">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockData10cacheLevelE"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockData10cacheLevelE"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData::cacheLevel__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData_1aae5cef5af5ba1a253e9df4b26174629a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheLevel</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData10cacheLevelE" title="Link to this definition"></a><br /></dt>
<dd><p>The cache level of the block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData8priorityE">
<span id="_CPPv3N12tensorrt_llm8executor22KVCacheStoredBlockData8priorityE"></span><span id="_CPPv2N12tensorrt_llm8executor22KVCacheStoredBlockData8priorityE"></span><span id="tensorrt_llm::executor::KVCacheStoredBlockData::priority__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredBlockData_1a8e2093038716a967af544a3c5e319629"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">priority</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockData8priorityE" title="Link to this definition"></a><br /></dt>
<dd><p>The priority of the block. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17KVCacheStoredDataE">
<span id="_CPPv3N12tensorrt_llm8executor17KVCacheStoredDataE"></span><span id="_CPPv2N12tensorrt_llm8executor17KVCacheStoredDataE"></span><span id="tensorrt_llm::executor::KVCacheStoredData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredData"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheStoredData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredDataE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17KVCacheStoredData10parentHashE">
<span id="_CPPv3N12tensorrt_llm8executor17KVCacheStoredData10parentHashE"></span><span id="_CPPv2N12tensorrt_llm8executor17KVCacheStoredData10parentHashE"></span><span id="tensorrt_llm::executor::KVCacheStoredData::parentHash__std::optional:IdType:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredData_1af93fe9da1a650d09d831591da6dbe913"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">parentHash</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredData10parentHashE" title="Link to this definition"></a><br /></dt>
<dd><p>The parent of this sequence of stored blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17KVCacheStoredData6blocksE">
<span id="_CPPv3N12tensorrt_llm8executor17KVCacheStoredData6blocksE"></span><span id="_CPPv2N12tensorrt_llm8executor17KVCacheStoredData6blocksE"></span><span id="tensorrt_llm::executor::KVCacheStoredData::blocks__std::vector:KVCacheStoredBlockData:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheStoredData_1a02f84a6a706d9baece7dbd0f104cdff4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KVCacheStoredBlockDataE" title="tensorrt_llm::executor::KVCacheStoredBlockData"><span class="n"><span class="pre">KVCacheStoredBlockData</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">blocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17KVCacheStoredData6blocksE" title="Link to this definition"></a><br /></dt>
<dd><p>A sequence of blocks. The parent of block <code class="docutils literal notranslate"><span class="pre">i</span></code> is block <code class="docutils literal notranslate"><span class="pre">i-1</span></code></p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedDataE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedDataE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedDataE"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheUpdatedData</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedDataE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData18KVCacheUpdatedDataE6IdType">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedData18KVCacheUpdatedDataE6IdType"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedData18KVCacheUpdatedDataE6IdType"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData::KVCacheUpdatedData__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData_1a5ac325874ac9863855823a798a88037c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KVCacheUpdatedData</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">blockHash</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData18KVCacheUpdatedDataE6IdType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData17cacheLevelUpdatedE10SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedData17cacheLevelUpdatedE10SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedData17cacheLevelUpdatedE10SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData::cacheLevelUpdated__SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData_1ad7fe06e5893b1ecf0fe4032b52855164"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedDataE" title="tensorrt_llm::executor::KVCacheUpdatedData"><span class="n"><span class="pre">KVCacheUpdatedData</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">cacheLevelUpdated</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">oldValue</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">newValue</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData17cacheLevelUpdatedE10SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData15priorityUpdatedE10SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedData15priorityUpdatedE10SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedData15priorityUpdatedE10SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData::priorityUpdated__SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData_1a62feae8ded6c060b17f5c54cb41a58e0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedDataE" title="tensorrt_llm::executor::KVCacheUpdatedData"><span class="n"><span class="pre">KVCacheUpdatedData</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">priorityUpdated</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">oldValue</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">newValue</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData15priorityUpdatedE10SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData9blockHashE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedData9blockHashE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedData9blockHashE"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData::blockHash__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData_1a5dc8d421acb9f870ee86307982f6c76b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">blockHash</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData9blockHashE" title="Link to this definition"></a><br /></dt>
<dd><p>The hash of the updated block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData10cacheLevelE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedData10cacheLevelE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedData10cacheLevelE"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData::cacheLevel__std::optional:KVCacheEventDiff:SizeType32::"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData_1a9c9779aca96e1a3e06994ad690c17f9a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE" title="tensorrt_llm::executor::KVCacheEventDiff"><span class="n"><span class="pre">KVCacheEventDiff</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheLevel</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData10cacheLevelE" title="Link to this definition"></a><br /></dt>
<dd><p>The updated value of the cacheLevel field. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData8priorityE">
<span id="_CPPv3N12tensorrt_llm8executor18KVCacheUpdatedData8priorityE"></span><span id="_CPPv2N12tensorrt_llm8executor18KVCacheUpdatedData8priorityE"></span><span id="tensorrt_llm::executor::KVCacheUpdatedData::priority__std::optional:KVCacheEventDiff:SizeType32::"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KVCacheUpdatedData_1a1ef3af844442752610874e4bddf88c06"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor16KVCacheEventDiffE" title="tensorrt_llm::executor::KVCacheEventDiff"><span class="n"><span class="pre">KVCacheEventDiff</span></span></a><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">priority</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18KVCacheUpdatedData8priorityE" title="Link to this definition"></a><br /></dt>
<dd><p>The updated value of the priority field. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfigE"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig25LogitsPostProcessorConfigENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEEb">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig25LogitsPostProcessorConfigENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEEb"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig25LogitsPostProcessorConfigENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEEb"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::LogitsPostProcessorConfig__std::optional:LogitsPostProcessorMap:.std::optional:LogitsPostProcessorBatched:.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a296cabed99e1093aa70de87520278d8d"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessorConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">processorMap</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">processorBatched</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">replicate</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig25LogitsPostProcessorConfigENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig15getProcessorMapEv">
<span id="_CPPv3NK12tensorrt_llm8executor25LogitsPostProcessorConfig15getProcessorMapEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25LogitsPostProcessorConfig15getProcessorMapEv"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::getProcessorMapC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a58f16d87581a027960f7a9a7b35e62f0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getProcessorMap</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig15getProcessorMapEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig19getProcessorBatchedEv">
<span id="_CPPv3NK12tensorrt_llm8executor25LogitsPostProcessorConfig19getProcessorBatchedEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25LogitsPostProcessorConfig19getProcessorBatchedEv"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::getProcessorBatchedC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a83f9ad50e26859cc6433867a536bd95b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getProcessorBatched</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig19getProcessorBatchedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig12getReplicateEv">
<span id="_CPPv3NK12tensorrt_llm8executor25LogitsPostProcessorConfig12getReplicateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25LogitsPostProcessorConfig12getReplicateEv"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::getReplicateC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1ae468e1f83b44da9a16b5e50e3b826b0f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getReplicate</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25LogitsPostProcessorConfig12getReplicateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig15setProcessorMapERK22LogitsPostProcessorMap">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig15setProcessorMapERK22LogitsPostProcessorMap"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig15setProcessorMapERK22LogitsPostProcessorMap"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::setProcessorMap__LogitsPostProcessorMapCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a16259a87bd2e42ae52bfe98f4f01ee6e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setProcessorMap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">processorMap</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig15setProcessorMapERK22LogitsPostProcessorMap" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig19setProcessorBatchedERK26LogitsPostProcessorBatched">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig19setProcessorBatchedERK26LogitsPostProcessorBatched"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig19setProcessorBatchedERK26LogitsPostProcessorBatched"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::setProcessorBatched__LogitsPostProcessorBatchedCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a39acb85e8c744c1657bc9e8d3b527c95"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setProcessorBatched</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">processorBatched</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig19setProcessorBatchedERK26LogitsPostProcessorBatched" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig12setReplicateEb">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig12setReplicateEb"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig12setReplicateEb"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::setReplicate__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a07a324ea77a570ed567c305fefc7c305"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setReplicate</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">replicate</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig12setReplicateEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig13mProcessorMapE">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig13mProcessorMapE"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig13mProcessorMapE"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::mProcessorMap__std::optional:LogitsPostProcessorMap:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a4123aa8b24b56aa7c9d3d45d9367dda1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mProcessorMap</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig13mProcessorMapE" title="Link to this definition"></a><br /></dt>
<dd><p>mapping from post processor names to non-batched post processors </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig17mProcessorBatchedE">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig17mProcessorBatchedE"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig17mProcessorBatchedE"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::mProcessorBatched__std::optional:LogitsPostProcessorBatched:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1a6fbb062ef3127599168b3ba7dc2d8089"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mProcessorBatched</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig17mProcessorBatchedE" title="Link to this definition"></a><br /></dt>
<dd><p>single batched post processor </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig10mReplicateE">
<span id="_CPPv3N12tensorrt_llm8executor25LogitsPostProcessorConfig10mReplicateE"></span><span id="_CPPv2N12tensorrt_llm8executor25LogitsPostProcessorConfig10mReplicateE"></span><span id="tensorrt_llm::executor::LogitsPostProcessorConfig::mReplicate__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LogitsPostProcessorConfig_1ac2be6c0331013174f435f6fb0db3adf5"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mReplicate</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25LogitsPostProcessorConfig10mReplicateE" title="Link to this definition"></a><br /></dt>
<dd><p>If set to true, logits post processor will run on all TP ranks in last PP rank. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfigE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::LookaheadDecodingConfig__SizeType32.SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1abef9e49b2e2e6312cf4c280146d24c0a"></span><span class="sig-name descname"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">windowSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ngramSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">verificationSetSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::LookaheadDecodingConfig"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a2affd5248374a476986c6aadc51eac4f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::eq-operator__LookaheadDecodingConfigCRC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a419b3ac66d9136151a2a0b7d526ad9b3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a1c0c7073a69264dc8a9391f019df8490"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">tuple</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">get</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getWindowSizeC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a8aa7577b18ff38bf4bc9640acf9aae47"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWindowSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getNgramSizeC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a5b20a75ca2a5e7f08259f0500330e58e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNgramSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getVerificationSetSizeC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1adad2572691ad3764b01c48b8ba2885c0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getVerificationSetSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::calculateSpeculativeResourceC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a703a176c65e6095b092b1b389d02cb9b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">tuple</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">calculateSpeculativeResource</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv" title="Link to this definition"></a><br /></dt>
<dd><p>return &lt;maxDecodingTokens, maxPathLen, maxDraftTokens, maxDraftPathLen&gt; </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::isLE__LookaheadDecodingConfigCRC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a87a081925bb740bf34f8c3a0a65f4180"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLE</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">that</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>return true when <code class="docutils literal notranslate"><span class="pre">this</span></code> can be executed on resources defined by <code class="docutils literal notranslate"><span class="pre">that</span></code></p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::isLegal__SizeType32.SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1aec6d6b6c329339fe92c04b3c99606c27"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLegal</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">windowSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ngramSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">verificationSetSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd><p>return true when the parameter combination is valid. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::mWindowSize__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a5325178fb1f33a47552ac21f4315e6fa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWindowSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::mNgramSize__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a47e578a6e17991ce817a1763cf610e57"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNgramSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::mVerificationSetSize__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a01deb81773d9c4ccf37074d773f5ca34"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVerificationSetSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfigE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfigE"></span><span id="tensorrt_llm::executor::LoraConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LoraConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for LoRA. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE"></span><span id="tensorrt_llm::executor::LoraConfig::LoraConfig__IdType.std::optional:Tensor:.std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a2a12dcb55bc38dd108c977c7b2fb4a55"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LoraConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">taskId</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">weights</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">config</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv"></span><span id="tensorrt_llm::executor::LoraConfig::getTaskIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1ad1d8cbbfcbd2046bf1b73ce608582c1d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTaskId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor10LoraConfig10getWeightsEv">
<span id="_CPPv3NK12tensorrt_llm8executor10LoraConfig10getWeightsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor10LoraConfig10getWeightsEv"></span><span id="tensorrt_llm::executor::LoraConfig::getWeightsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1af3f46ffd3779842a130a0f658edcdd6f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWeights</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig10getWeightsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor10LoraConfig9getConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor10LoraConfig9getConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor10LoraConfig9getConfigEv"></span><span id="tensorrt_llm::executor::LoraConfig::getConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a17cb6307358a40303326dddcaf2971d3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig9getConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig7mTaskIdE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig7mTaskIdE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig7mTaskIdE"></span><span id="tensorrt_llm::executor::LoraConfig::mTaskId__IdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a863ffa00f268192c572d4e3f0737d535"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTaskId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig7mTaskIdE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora task id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig8mWeightsE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig8mWeightsE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig8mWeightsE"></span><span id="tensorrt_llm::executor::LoraConfig::mWeights__std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1ae6941906a80853143186df62f10d8d95"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWeights</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig8mWeightsE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora weights. See TRT-LLM documentation for expected shapes and types. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig7mConfigE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig7mConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig7mConfigE"></span><span id="tensorrt_llm::executor::LoraConfig::mConfig__std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a70553cf896bdbb10718070e7da355bbc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig7mConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora configuration. See TRT-LLM documentation for detailed description of the config tensor. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11MropeConfigE">
<span id="_CPPv3N12tensorrt_llm8executor11MropeConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor11MropeConfigE"></span><span id="tensorrt_llm::executor::MropeConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1MropeConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MropeConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for mrope. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11MropeConfig11MropeConfigE6Tensor10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor11MropeConfig11MropeConfigE6Tensor10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor11MropeConfig11MropeConfigE6Tensor10SizeType32"></span><span id="tensorrt_llm::executor::MropeConfig::MropeConfig__Tensor.SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1MropeConfig_1a163652c73e6c583fc583dbd3afb74645"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MropeConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">mropeRoratySinCos</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">mropePositionDeltas</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11MropeConfig11MropeConfigE6Tensor10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11MropeConfig20getMRopeRotarySinCosEv">
<span id="_CPPv3NK12tensorrt_llm8executor11MropeConfig20getMRopeRotarySinCosEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11MropeConfig20getMRopeRotarySinCosEv"></span><span id="tensorrt_llm::executor::MropeConfig::getMRopeRotarySinCosC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1MropeConfig_1a7656f17ce01a559bcb46ea4999ab6298"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMRopeRotarySinCos</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11MropeConfig20getMRopeRotarySinCosEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor11MropeConfig22getMRopePositionDeltasEv">
<span id="_CPPv3NK12tensorrt_llm8executor11MropeConfig22getMRopePositionDeltasEv"></span><span id="_CPPv2NK12tensorrt_llm8executor11MropeConfig22getMRopePositionDeltasEv"></span><span id="tensorrt_llm::executor::MropeConfig::getMRopePositionDeltasC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1MropeConfig_1a5cc9b41eac1cc0bee6ffdd940357f056"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMRopePositionDeltas</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor11MropeConfig22getMRopePositionDeltasEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11MropeConfig18mMRopeRotarySinCosE">
<span id="_CPPv3N12tensorrt_llm8executor11MropeConfig18mMRopeRotarySinCosE"></span><span id="_CPPv2N12tensorrt_llm8executor11MropeConfig18mMRopeRotarySinCosE"></span><span id="tensorrt_llm::executor::MropeConfig::mMRopeRotarySinCos__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1MropeConfig_1abfd4a93aca9c91e294126d6b7d225734"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMRopeRotarySinCos</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11MropeConfig18mMRopeRotarySinCosE" title="Link to this definition"></a><br /></dt>
<dd><p>The mrope rotary sin and cos cache. Expected shape: [maxPositionEmbeddings*rotaryEmbeddingDim],Data type must float32. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11MropeConfig20mMRopePositionDeltasE">
<span id="_CPPv3N12tensorrt_llm8executor11MropeConfig20mMRopePositionDeltasE"></span><span id="_CPPv2N12tensorrt_llm8executor11MropeConfig20mMRopePositionDeltasE"></span><span id="tensorrt_llm::executor::MropeConfig::mMRopePositionDeltas__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1MropeConfig_1a34c4fbc4d6a6c1ee7d1a407c0596546b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMRopePositionDeltas</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11MropeConfig20mMRopePositionDeltasE" title="Link to this definition"></a><br /></dt>
<dd><p>The mrope position deltas. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfigE"></span><span id="tensorrt_llm::executor::OrchestratorConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OrchestratorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEEb">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEEb"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEEb"></span><span id="tensorrt_llm::executor::OrchestratorConfig::OrchestratorConfig__b.ss.std::shared_ptr:mpi::MpiComm:.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a48c20f133f76c052ad8d9174462648f4"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OrchestratorConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">isOrchestrator</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">workerExecutablePath</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;&quot;</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">orchLeaderComm</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">spawnProcesses</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getIsOrchestratorC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a0faa15a0f3ce88cb25eb07a1e1a91d34"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getIsOrchestrator</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getWorkerExecutablePathC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a603abfa699a11b7e910278cc9c027542"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWorkerExecutablePath</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getOrchLeaderCommC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1adeb40968afe900fff60a0ee58e500cff"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOrchLeaderComm</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getSpawnProcessesEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig17getSpawnProcessesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig17getSpawnProcessesEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getSpawnProcessesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a1b3409f1355ea612ac0c1fdae0c05ded"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSpawnProcesses</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getSpawnProcessesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setIsOrchestrator__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a5384e155759356518a2bca388a1ae1d0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setIsOrchestrator</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">isOrchestrator</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setWorkerExecutablePath__ssCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a557913170113b0f37daab422bb8552d1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setWorkerExecutablePath</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">workerExecutablePath</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setOrchLeaderComm__std::shared_ptr:mpi::MpiComm:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1ab4e88541c7a38d3675485f227030f7e2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOrchLeaderComm</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchLeaderComm</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setSpawnProcessesEb">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig17setSpawnProcessesEb"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig17setSpawnProcessesEb"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setSpawnProcesses__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a32a045e475fb0515f1959022b1d56ba1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSpawnProcesses</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">spawnProcesses</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setSpawnProcessesEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mIsOrchestrator__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a272f1d1692fae68ed8242d8180326977"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mIsOrchestrator</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mWorkerExecutablePath__ss"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a8a4c20267c20632d8107e9b8041198c2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWorkerExecutablePath</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mOrchLeaderComm__std::shared_ptr:mpi::MpiComm:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a16ac5a458ad5e4e048d0badcba4b54d5"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOrchLeaderComm</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mSpawnProcessesE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig15mSpawnProcessesE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig15mSpawnProcessesE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mSpawnProcesses__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1af4f62bc30dd01bfd0ac44b7dfd62323d"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSpawnProcesses</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mSpawnProcessesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfigE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfigE"></span><span id="tensorrt_llm::executor::OutputConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OutputConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration that controls the outputs of a <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a>. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbbb">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbbb"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbbb"></span><span id="tensorrt_llm::executor::OutputConfig::OutputConfig__b.b.b.b.b.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a8a4a66ff709fcf387de53e98785970ab"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OutputConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnLogProbs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnContextLogits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnGenerationLogits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">excludeInputFromOutput</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnEncoderOutput</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnPerfMetrics</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbbb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig14returnLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig14returnLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig14returnLogProbsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a1f23a994b65629c8beee0bbb79f71ee7"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig14returnLogProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain log probabilities. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnContextLogits__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a4419de36cc2f99f8c4ce8911ee0dadf5"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnContextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain the context logits. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnGenerationLogits__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a1d8dda4081c539f18f32af0be70c0f17"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnGenerationLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain the generation logits. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE"></span><span id="tensorrt_llm::executor::OutputConfig::excludeInputFromOutput__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1ab38293c37f6f6b60e3e39f68f18f7b90"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">excludeInputFromOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if output tokens in <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should include the input tokens. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE"></span><span id="tensorrt_llm::executor::OutputConfig::returnEncoderOutput__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a11049bc5a1b255c703533515539ec66a"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnEncoderOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain encoder output hidden states (for encoder-only and encoder-decoder models). Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig17returnPerfMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig17returnPerfMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig17returnPerfMetricsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnPerfMetrics__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a7d6aef80893aa3eabbc78f0a504f8e1a"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnPerfMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig17returnPerfMetricsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain performance metrics. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfigE"></span><span id="tensorrt_llm::executor::ParallelConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ParallelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>A configuration class for the parallel execution parameters Currently only supports commType = CommunicationType::kMPI. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE"></span><span id="tensorrt_llm::executor::ParallelConfig::ParallelConfig__CommunicationType.CommunicationMode.std::optional:std::vector:SizeType32::.std::optional:std::vector:SizeType32::.std::optional:OrchestratorConfig:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1af2a7d554738de524f3e0207d4290d30f"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ParallelConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">commType</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE" title="tensorrt_llm::executor::CommunicationType::kMPI"><span class="n"><span class="pre">kMPI</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">commMode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE" title="tensorrt_llm::executor::CommunicationMode::kLEADER"><span class="n"><span class="pre">kLEADER</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">deviceIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">participantIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE" title="Link to this definition"></a><br /></dt>
<dd><p>Constructor. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>commType</strong> – The communication type. See CommunicationType. </p></li>
<li><p><strong>commMode</strong> – The communication mode. See CommunicationMode. </p></li>
<li><p><strong>deviceIds</strong> – The IDs of the GPUs involved in the execution of the model </p></li>
<li><p><strong>participantIds</strong> – The participant IDs (MPI ranks if commType == kMPI) involved in the execution of the model. The first participant is considered to be the leader. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getCommunicationTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ad384d17a72f351d994e604594d6a6f4b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCommunicationType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getCommunicationModeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a9bf12e33fc86542ee77ff1488fa7835c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCommunicationMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getDeviceIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a4c0de3df9bae6b2261f9b65df7a07da9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDeviceIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getParticipantIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a5c24e64fa629a2291c11959faa03cd3a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParticipantIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getOrchestratorConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a51b3d7fe63d40e0e99d06bd04df19aa9"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOrchestratorConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType"></span><span id="tensorrt_llm::executor::ParallelConfig::setCommunicationType__CommunicationType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ad16a96f04fa3474bf67452910f64a8bc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCommunicationType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode"></span><span id="tensorrt_llm::executor::ParallelConfig::setCommunicationMode__CommunicationMode"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a1bb2819d5c1add8908d9db259b16c741"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCommunicationMode</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">mode</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE"></span><span id="tensorrt_llm::executor::ParallelConfig::setDeviceIds__std::vector:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a6bcd6e1acdde379946d599e884fb7ee1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDeviceIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">deviceIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE"></span><span id="tensorrt_llm::executor::ParallelConfig::setParticipantIds__std::vector:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a7e3f6c11ef029cdee705305ca1129a0a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setParticipantIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">participantIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig"></span><span id="tensorrt_llm::executor::ParallelConfig::setOrchestratorConfig__OrchestratorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a72057d9d07087fc7b6fdb2795bf1fb5e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOrchestratorConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommTypeE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig9mCommTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig9mCommTypeE"></span><span id="tensorrt_llm::executor::ParallelConfig::mCommType__CommunicationType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ae1b162dc41f8541aba52d90c6bd76d18"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The type of communication protocol used. Default is MPI. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommModeE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig9mCommModeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig9mCommModeE"></span><span id="tensorrt_llm::executor::ParallelConfig::mCommMode__CommunicationMode"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ad5b06cc50716576658e25aa7d30fd6b6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommModeE" title="Link to this definition"></a><br /></dt>
<dd><p>The mode of communication. See CommunicationMode. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE"></span><span id="tensorrt_llm::executor::ParallelConfig::mDeviceIds__std::optional:std::vector:SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a4a9207031d67fbe267d38d89c8901e81"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDeviceIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The GPU device ids to use for executing this model. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE"></span><span id="tensorrt_llm::executor::ParallelConfig::mParticipantIds__std::optional:std::vector:SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ae0e3d3936b187b3fed26b16220ee8a57"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mParticipantIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The participant ids (MPI ranks for example) used for executing this model. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE"></span><span id="tensorrt_llm::executor::ParallelConfig::mOrchestratorConfig__std::optional:OrchestratorConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a0a8ccedf7de365ca80be267e90627954"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOrchestratorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Optional orchestrator configuration. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfigE"></span><span id="tensorrt_llm::executor::PeftCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PeftCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>config for PeftCacheManager </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::PeftCacheConfig__SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.std::optional:float:CR.std::optional:s:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a8e8b013299c4a6bfce417b75f3b7b819"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PeftCacheConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numHostModuleLayer</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numDeviceModuleLayer</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">optimalAdapterSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig26kDefaultOptimalAdapterSizeE" title="tensorrt_llm::executor::PeftCacheConfig::kDefaultOptimalAdapterSize"><span class="n"><span class="pre">kDefaultOptimalAdapterSize</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxAdapterSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig22kDefaultMaxAdapterSizeE" title="tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxAdapterSize"><span class="n"><span class="pre">kDefaultMaxAdapterSize</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numPutWorkers</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numEnsureWorkers</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numCopyStreams</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxPagesPerBlockHost</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig28kDefaultMaxPagesPerBlockHostE" title="tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxPagesPerBlockHost"><span class="n"><span class="pre">kDefaultMaxPagesPerBlockHost</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxPagesPerBlockDevice</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig30kDefaultMaxPagesPerBlockDeviceE" title="tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxPagesPerBlockDevice"><span class="n"><span class="pre">kDefaultMaxPagesPerBlockDevice</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">deviceCachePercent</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">hostCacheSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig"></span><span id="tensorrt_llm::executor::PeftCacheConfig::eq-operator__PeftCacheConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ab60292786cb2f414b52a8d571708a281"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumHostModuleLayerC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ad0a03ff1b983f8c306912156a370d436"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumHostModuleLayer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumDeviceModuleLayerC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ac6585fa80da031f50cfd241dab5516d4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumDeviceModuleLayer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getOptimalAdapterSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a550e64682974fb60078a5c858d84bcc2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOptimalAdapterSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getMaxAdapterSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a7e11e64fc78e41f098df0a9b2d683939"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxAdapterSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumPutWorkersC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1af92e38b38cb61de1dad2210a76011163"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumPutWorkers</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumEnsureWorkersC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abcaf1eefd4d9ef5fbffa47eb885b1b6d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumEnsureWorkers</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumCopyStreamsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a3e28ede570906ef9c3ccfc2898e0e4aa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumCopyStreams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getMaxPagesPerBlockHostC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1af7ab8d350fee59f9890e048393bf5a02"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxPagesPerBlockHost</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getMaxPagesPerBlockDeviceC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1aa7147fa1d4bbef5ce8cc038b5434cc8e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxPagesPerBlockDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getDeviceCachePercentC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a5550bf7fe17f230b013e2e754633d80c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDeviceCachePercent</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getHostCacheSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abc33e84d2e2764a885fa326814c3a4c6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getHostCacheSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig26kDefaultOptimalAdapterSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig26kDefaultOptimalAdapterSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig26kDefaultOptimalAdapterSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::kDefaultOptimalAdapterSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a669e2b2b2959e186fd9a23a1804f2373"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultOptimalAdapterSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig26kDefaultOptimalAdapterSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig22kDefaultMaxAdapterSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig22kDefaultMaxAdapterSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig22kDefaultMaxAdapterSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxAdapterSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1aedca3e5c48f032a321e6e3edaadc2d5e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultMaxAdapterSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">64</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig22kDefaultMaxAdapterSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig28kDefaultMaxPagesPerBlockHostE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig28kDefaultMaxPagesPerBlockHostE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig28kDefaultMaxPagesPerBlockHostE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxPagesPerBlockHost__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a85227290ca63d58758e17ef05368fff2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultMaxPagesPerBlockHost</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">24</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig28kDefaultMaxPagesPerBlockHostE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig30kDefaultMaxPagesPerBlockDeviceE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig30kDefaultMaxPagesPerBlockDeviceE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig30kDefaultMaxPagesPerBlockDeviceE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::kDefaultMaxPagesPerBlockDevice__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a8f3b9a6868ff0b03e63e37d98e108839"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultMaxPagesPerBlockDevice</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig30kDefaultMaxPagesPerBlockDeviceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumHostModuleLayer__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ae3a957e6c10ff32fda9cdfa261307a23"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumHostModuleLayer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumDeviceModuleLayer__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a6c734def4b320c40ec7758a2f9f2f928"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumDeviceModuleLayer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mOptimalAdapterSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a4cbcea2497e456f7f5c6988d9afba569"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOptimalAdapterSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mMaxAdapterSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abe38d2b75a813f5b2e73e5800083984a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxAdapterSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumPutWorkers__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ab5b4bc29cc88c7cc3dd7f90bd8632966"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumPutWorkers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumEnsureWorkers__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a0e10d459eaf84adcd6d50926775f5a45"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumEnsureWorkers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumCopyStreams__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a2d7ccbe5d9c11d02cab32a1a7557c511"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumCopyStreams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mMaxPagesPerBlockHost__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a968719a3d4bec709899c4bbb50e84a6f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxPagesPerBlockHost</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mMaxPagesPerBlockDevice__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abd1029f5855b119ea0b6c66c46e59eee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxPagesPerBlockDevice</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mDeviceCachePercent__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a838668e3174dd46dddb8bb60a9da0e05"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDeviceCachePercent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mHostCacheSize__std::optional:s:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a656930138cb6f162e7fc15c82778d1cc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mHostCacheSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfigE"></span><span id="tensorrt_llm::executor::PromptTuningConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for prompt tuning. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6TensorNSt8optionalI16VecTokenExtraIdsEE">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6TensorNSt8optionalI16VecTokenExtraIdsEE"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6TensorNSt8optionalI16VecTokenExtraIdsEE"></span><span id="tensorrt_llm::executor::PromptTuningConfig::PromptTuningConfig__Tensor.std::optional:VecTokenExtraIds:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1a5ef30446f49901fe54cf7d539be9a158"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">embeddingTable</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16VecTokenExtraIdsE" title="tensorrt_llm::executor::VecTokenExtraIds"><span class="n"><span class="pre">VecTokenExtraIds</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">inputTokenExtraIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6TensorNSt8optionalI16VecTokenExtraIdsEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv">
<span id="_CPPv3NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv"></span><span id="tensorrt_llm::executor::PromptTuningConfig::getEmbeddingTableC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1af8534fb4246a662291be6f7e17655ca0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEmbeddingTable</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig21getInputTokenExtraIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor18PromptTuningConfig21getInputTokenExtraIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18PromptTuningConfig21getInputTokenExtraIdsEv"></span><span id="tensorrt_llm::executor::PromptTuningConfig::getInputTokenExtraIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1ad7693f426c620eec280bf66202e22b34"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16VecTokenExtraIdsE" title="tensorrt_llm::executor::VecTokenExtraIds"><span class="n"><span class="pre">VecTokenExtraIds</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getInputTokenExtraIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig21getInputTokenExtraIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE"></span><span id="tensorrt_llm::executor::PromptTuningConfig::mEmbeddingTable__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1a4f6be4050b7b94a5950234e70b9df919"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEmbeddingTable</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE" title="Link to this definition"></a><br /></dt>
<dd><p>The prompt embedding table. Expected shape: [task vocab_size, hidden_size]. Data type must match model weights. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfig19mInputTokenExtraIdsE">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfig19mInputTokenExtraIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfig19mInputTokenExtraIdsE"></span><span id="tensorrt_llm::executor::PromptTuningConfig::mInputTokenExtraIds__std::optional:VecTokenExtraIds:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1a4c2d7366ec21819f8b579f73a1b5154b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16VecTokenExtraIdsE" title="tensorrt_llm::executor::VecTokenExtraIds"><span class="n"><span class="pre">VecTokenExtraIds</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mInputTokenExtraIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig19mInputTokenExtraIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The input token extra ids for KV Cache reuse when p-tuning is enabled. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestE">
<span id="_CPPv3N12tensorrt_llm8executor7RequestE"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestE"></span><span id="tensorrt_llm::executor::Request"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>A class that holds information about the request. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI11MropeConfigEENSt8optionalI10LoraConfigEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI22KvCacheRetentionConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEENSt8optionalI6IdTypeEEb12PriorityType11RequestTypeNSt8optionalI18ContextPhaseParamsEENSt8optionalI6TensorEENSt8optionalI10SizeType32EENSt8optionalI6TensorEE10SizeType32NSt8optionalI11EagleConfigEENSt8optionalI6TensorEENSt8optionalI20GuidedDecodingParamsEENSt8optionalI16MillisecondsTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI11MropeConfigEENSt8optionalI10LoraConfigEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI22KvCacheRetentionConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEENSt8optionalI6IdTypeEEb12PriorityType11RequestTypeNSt8optionalI18ContextPhaseParamsEENSt8optionalI6TensorEENSt8optionalI10SizeType32EENSt8optionalI6TensorEE10SizeType32NSt8optionalI11EagleConfigEENSt8optionalI6TensorEENSt8optionalI20GuidedDecodingParamsEENSt8optionalI16MillisecondsTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI11MropeConfigEENSt8optionalI10LoraConfigEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI22KvCacheRetentionConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEENSt8optionalI6IdTypeEEb12PriorityType11RequestTypeNSt8optionalI18ContextPhaseParamsEENSt8optionalI6TensorEENSt8optionalI10SizeType32EENSt8optionalI6TensorEE10SizeType32NSt8optionalI11EagleConfigEENSt8optionalI6TensorEENSt8optionalI20GuidedDecodingParamsEENSt8optionalI16MillisecondsTypeEE"></span><span id="tensorrt_llm::executor::Request::Request__VecTokens.SizeType32.b.SamplingConfigCR.OutputConfigCR.std::optional:SizeType32:CR.std::optional:SizeType32:CR.std::optional:std::vector:SizeType32::.std::optional:std::list:VecTokens::.std::optional:std::list:VecTokens::.std::optional:Tensor:.std::optional:ExternalDraftTokensConfig:.std::optional:PromptTuningConfig:.std::optional:MropeConfig:.std::optional:LoraConfig:.std::optional:LookaheadDecodingConfig:.std::optional:KvCacheRetentionConfig:.std::optional:ss:.std::optional:VecTokens:.std::optional:IdType:.b.PriorityType.RequestType.std::optional:ContextPhaseParams:.std::optional:Tensor:.std::optional:SizeType32:.std::optional:Tensor:.SizeType32.std::optional:EagleConfig:.std::optional:Tensor:.std::optional:GuidedDecodingParams:.std::optional:MillisecondsType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ac2bfed0a3d807e8faddbbb1d1a82c320"></span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">inputTokenIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokens</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">streaming</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">endId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">padId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">positionIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">badWords</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stopWords</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">embeddingBias</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">externalDraftTokensConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pTuningConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="tensorrt_llm::executor::MropeConfig"><span class="n"><span class="pre">MropeConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">mRopeConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">loraConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">lookaheadConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">kvCacheRetentionConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsPostProcessorName</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encoderInputTokenIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">clientId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnAllGeneratedTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12PriorityTypeE" title="tensorrt_llm::executor::PriorityType"><span class="n"><span class="pre">PriorityType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">priority</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request16kDefaultPriorityE" title="tensorrt_llm::executor::Request::kDefaultPriority"><span class="n"><span class="pre">kDefaultPriority</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestTypeE" title="tensorrt_llm::executor::RequestType"><span class="n"><span class="pre">RequestType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestTypeE" title="tensorrt_llm::executor::RequestType"><span class="n"><span class="pre">RequestType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestType35REQUEST_TYPE_CONTEXT_AND_GENERATIONE" title="tensorrt_llm::executor::RequestType::REQUEST_TYPE_CONTEXT_AND_GENERATION"><span class="n"><span class="pre">REQUEST_TYPE_CONTEXT_AND_GENERATION</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">contextPhaseParams</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encoderInputFeatures</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encoderOutputLength</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">crossAttentionMask</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numReturnSequences</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">eagleConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">skipCrossAttnBlocks</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">guidedDecodingParams</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16MillisecondsTypeE" title="tensorrt_llm::executor::MillisecondsType"><span class="n"><span class="pre">MillisecondsType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">allottedTimeMs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI11MropeConfigEENSt8optionalI10LoraConfigEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI22KvCacheRetentionConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEENSt8optionalI6IdTypeEEb12PriorityType11RequestTypeNSt8optionalI18ContextPhaseParamsEENSt8optionalI6TensorEENSt8optionalI10SizeType32EENSt8optionalI6TensorEE10SizeType32NSt8optionalI11EagleConfigEENSt8optionalI6TensorEENSt8optionalI20GuidedDecodingParamsEENSt8optionalI16MillisecondsTypeEE" title="Link to this definition"></a><br /></dt>
<dd><p>The <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> constructor. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>inputTokenIds</strong> – The input token ids </p></li>
<li><p><strong>maxTokens</strong> – The maximum number of tokens to generate </p></li>
<li><p><strong>streaming</strong> – Indicates if the responses should be streamed or not. Default is false. </p></li>
<li><p><strong>samplingConfig</strong> – The sampling configuration </p></li>
<li><p><strong>outputConfig</strong> – The output configuration </p></li>
<li><p><strong>endId</strong> – The end token id </p></li>
<li><p><strong>padId</strong> – The pad token id </p></li>
<li><p><strong>positionIds</strong> – The input position ids </p></li>
<li><p><strong>badWords</strong> – A list of bad words tokens. Each “word” can be composed of multiple tokens </p></li>
<li><p><strong>stopWords</strong> – A list of stop words tokens. Each “word” can be composed of multiple tokens </p></li>
<li><p><strong>embeddingBias</strong> – The embedding bias tensor. Expected type is kFP32 and shape is [vocab_size] </p></li>
<li><p><strong>externalDraftTokensConfig</strong> – The speculative decoding with external draft tokens configuration </p></li>
<li><p><strong>pTuningConfig</strong> – The prompt tuning configuration </p></li>
<li><p><strong>loraConfig</strong> – The LoRA configuration </p></li>
<li><p><strong>lookaheadConfig</strong> – The lookahead speculative decoding configuration </p></li>
<li><p><strong>logitsPostProcessorName</strong> – The logits postprocessor name. Must correspond to one of the logits postprocessor </p></li>
<li><p><strong>kvCacheRetentionConfig</strong> – The configuration used for KV cache block eviction. name provided to the <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1ExecutorConfig"><span class="std std-ref">ExecutorConfig</span></a>. </p></li>
<li><p><strong>encoderInputTokenIds</strong> – The encoder input token ids for encoder-decoder models, or encoder-only models </p></li>
<li><p><strong>returnAllGeneratedTokens</strong> – Indicates whether to return the full beams or just the newly generated tokens after every streaming step. </p></li>
<li><p><strong>priority</strong> – Sets the execution priority of this request. </p></li>
<li><p><strong>encoderInputFeatures</strong> – Encoder input features for multimodal models. </p></li>
<li><p><strong>encoderOutputLength</strong> – Encoder output length if encoder input and output have different lengths (due to convolution down-sampling, etc.) </p></li>
<li><p><strong>crossAttentionMask</strong> – Cross attention mask. </p></li>
<li><p><strong>type</strong> – Indicate the request type for disaggregated serving mode. </p></li>
<li><p><strong>contextPhaseParams</strong> – Generated token ID from context only executor. </p></li>
<li><p><strong>numReturnSequences</strong> – The number of returning sequences. </p></li>
<li><p><strong>eagleConfig</strong> – The EAGLE speculative decoding configuration </p></li>
<li><p><strong>skipCrossAttnBlocks</strong> – Skip the cross attention transformer blocks or not. </p></li>
<li><p><strong>guidedDecodingParams</strong> – The guided decoding parameters. </p></li>
<li><p><strong>allottedTimeMs</strong> – The allotted time in milliseconds after which the request is finished with a timedOut finish reason. The request always will exceed this time slightly, but at most with 1 forward pass. A request can be timed-out before ever being scheduled. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor7Request7RequestERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7Request7RequestERK7Request"></span><span id="tensorrt_llm::executor::Request::Request__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ac1ba647ff89d0de162cb6b7e8a3b3218"></span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request" title="tensorrt_llm::executor::Request::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request">
<span id="_CPPv3N12tensorrt_llm8executor7Request7RequestERR7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7Request7RequestERR7Request"></span><span id="tensorrt_llm::executor::Request::Request__RequestRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aea6f0d55747d21a96b77e164de0dcacb"></span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request" title="tensorrt_llm::executor::Request::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestaSERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor7RequestaSERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestaSERK7Request"></span><span id="tensorrt_llm::executor::Request::assign-operator__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad465314b462d06a4c1c4bd90d7ae35bb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestaSERK7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestaSERR7Request">
<span id="_CPPv3N12tensorrt_llm8executor7RequestaSERR7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestaSERR7Request"></span><span id="tensorrt_llm::executor::Request::assign-operator__RequestRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a1702e650276cd70a43e88302ea24f815"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestaSERR7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor7RequestD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestD0Ev"></span><span id="tensorrt_llm::executor::Request::~Request"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad615bd9f0869e60c95bb9dcb01ed8ed6"></span><span class="sig-name descname"><span class="n"><span class="pre">~Request</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request16getInputTokenIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request16getInputTokenIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request16getInputTokenIdsEv"></span><span id="tensorrt_llm::executor::Request::getInputTokenIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae5023d8d2d8cfa9bd3e3a16c76cc6819"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getInputTokenIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request16getInputTokenIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request12getMaxTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request12getMaxTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request12getMaxTokensEv"></span><span id="tensorrt_llm::executor::Request::getMaxTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a20edd026c107c8e66ba8e44b141cf8f1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request12getMaxTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request15getMaxNewTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request15getMaxNewTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request15getMaxNewTokensEv"></span><span id="tensorrt_llm::executor::Request::getMaxNewTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a357aeecb46381de018ddd3836b1f23d3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxNewTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request15getMaxNewTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request12getStreamingEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request12getStreamingEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request12getStreamingEv"></span><span id="tensorrt_llm::executor::Request::getStreamingC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a04fe14eab1bf2031220dfce904f6e736"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getStreaming</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request12getStreamingEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request17getSamplingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request17getSamplingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request17getSamplingConfigEv"></span><span id="tensorrt_llm::executor::Request::getSamplingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae89bc1ae8e0a108e2f74e43470631e4a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSamplingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request17getSamplingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request15getOutputConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request15getOutputConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request15getOutputConfigEv"></span><span id="tensorrt_llm::executor::Request::getOutputConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a4fe15a7d4be498a08b8dcbe999a4241d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request15getOutputConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request8getEndIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request8getEndIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request8getEndIdEv"></span><span id="tensorrt_llm::executor::Request::getEndIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a0ef03baab26435b1a0f52fb17921871d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEndId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request8getEndIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request8getPadIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request8getPadIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request8getPadIdEv"></span><span id="tensorrt_llm::executor::Request::getPadIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aeb2c8f786eefb6cb288f535ae0f8c38f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPadId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request8getPadIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request14getPositionIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request14getPositionIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request14getPositionIdsEv"></span><span id="tensorrt_llm::executor::Request::getPositionIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a438538a90ed116c31ba77134b891be19"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPositionIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request14getPositionIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request11getBadWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request11getBadWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request11getBadWordsEv"></span><span id="tensorrt_llm::executor::Request::getBadWordsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aa2203882fac60bc972d3190f5a952cbf"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBadWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request11getBadWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request12getStopWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request12getStopWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request12getStopWordsEv"></span><span id="tensorrt_llm::executor::Request::getStopWordsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a0ed1ed7425f09806e0d42373ba0b1a23"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getStopWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request12getStopWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv"></span><span id="tensorrt_llm::executor::Request::getEmbeddingBiasC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aa090ed0051151568052c4b8a662ac38e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEmbeddingBias</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv"></span><span id="tensorrt_llm::executor::Request::getExternalDraftTokensConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a575e5568daa7b708d2e49d3c0e27f4ca"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv"></span><span id="tensorrt_llm::executor::Request::getPromptTuningConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1add08e33a6e9719b85ab0ddfb288c62d2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPromptTuningConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request14getMropeConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request14getMropeConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request14getMropeConfigEv"></span><span id="tensorrt_llm::executor::Request::getMropeConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a2b984d67f3eb7f4a73044416f615fb8f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="tensorrt_llm::executor::MropeConfig"><span class="n"><span class="pre">MropeConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMropeConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request14getMropeConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request13getLoraConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request13getLoraConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request13getLoraConfigEv"></span><span id="tensorrt_llm::executor::Request::getLoraConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a358e77b87c918210c398bf78a98cafda"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLoraConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request13getLoraConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request18getLookaheadConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request18getLookaheadConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request18getLookaheadConfigEv"></span><span id="tensorrt_llm::executor::Request::getLookaheadConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a1380dab44644b7b7b267d6b3309fba01"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLookaheadConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request18getLookaheadConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request25getKvCacheRetentionConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request25getKvCacheRetentionConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request25getKvCacheRetentionConfigEv"></span><span id="tensorrt_llm::executor::Request::getKvCacheRetentionConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a8bc8c18d0bf4ee04c1b0df6e0f497ec7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getKvCacheRetentionConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request25getKvCacheRetentionConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv"></span><span id="tensorrt_llm::executor::Request::getLogitsPostProcessorNameC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a29a2a56fef482187e36fcb41238f238f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogitsPostProcessorName</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv"></span><span id="tensorrt_llm::executor::Request::getEncoderInputTokenIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a31dcafcde78f9af38928b038dfd7e8a8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEncoderInputTokenIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request11getClientIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request11getClientIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request11getClientIdEv"></span><span id="tensorrt_llm::executor::Request::getClientIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a5b78ea8903ede19e5a2991128146a40c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getClientId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request11getClientIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request11getPriorityEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request11getPriorityEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request11getPriorityEv"></span><span id="tensorrt_llm::executor::Request::getPriorityC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ace036f9c39b9ec797146b67b97c7f454"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12PriorityTypeE" title="tensorrt_llm::executor::PriorityType"><span class="n"><span class="pre">PriorityType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPriority</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request11getPriorityEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request27getReturnAllGeneratedTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request27getReturnAllGeneratedTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request27getReturnAllGeneratedTokensEv"></span><span id="tensorrt_llm::executor::Request::getReturnAllGeneratedTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a9469704f8ae0d13943fc85cc55473893"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getReturnAllGeneratedTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request27getReturnAllGeneratedTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request21getContextPhaseParamsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request21getContextPhaseParamsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request21getContextPhaseParamsEv"></span><span id="tensorrt_llm::executor::Request::getContextPhaseParamsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a6a81263006f5f937f9692e8da3e6965c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getContextPhaseParams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request21getContextPhaseParamsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputFeaturesEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request23getEncoderInputFeaturesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request23getEncoderInputFeaturesEv"></span><span id="tensorrt_llm::executor::Request::getEncoderInputFeaturesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ab006fa4cfd9d17df1517ccf26c958ef7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEncoderInputFeatures</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputFeaturesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request22getEncoderOutputLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request22getEncoderOutputLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request22getEncoderOutputLengthEv"></span><span id="tensorrt_llm::executor::Request::getEncoderOutputLengthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1adb813f19b1b1d66456b412561166d0ab"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEncoderOutputLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request22getEncoderOutputLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request21getCrossAttentionMaskEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request21getCrossAttentionMaskEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request21getCrossAttentionMaskEv"></span><span id="tensorrt_llm::executor::Request::getCrossAttentionMaskC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad3c31467a350829ab39b23402d4ca443"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCrossAttentionMask</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request21getCrossAttentionMaskEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request14getRequestTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request14getRequestTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request14getRequestTypeEv"></span><span id="tensorrt_llm::executor::Request::getRequestTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a1006ee8f28e9abaec817bba470f6be6a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestTypeE" title="tensorrt_llm::executor::RequestType"><span class="n"><span class="pre">RequestType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRequestType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request14getRequestTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request21getNumReturnSequencesEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request21getNumReturnSequencesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request21getNumReturnSequencesEv"></span><span id="tensorrt_llm::executor::Request::getNumReturnSequencesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a39578a9fbd1b299a7567572c95e6a0b0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumReturnSequences</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request21getNumReturnSequencesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request14getEagleConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request14getEagleConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request14getEagleConfigEv"></span><span id="tensorrt_llm::executor::Request::getEagleConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a8586e97f346691fb5a3cb04917fb2c49"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEagleConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request14getEagleConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request22getSkipCrossAttnBlocksEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request22getSkipCrossAttnBlocksEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request22getSkipCrossAttnBlocksEv"></span><span id="tensorrt_llm::executor::Request::getSkipCrossAttnBlocksC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a530aa5a065cd0c88f8fbfdfdf9fc58f6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSkipCrossAttnBlocks</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request22getSkipCrossAttnBlocksEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request23getGuidedDecodingParamsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request23getGuidedDecodingParamsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request23getGuidedDecodingParamsEv"></span><span id="tensorrt_llm::executor::Request::getGuidedDecodingParamsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a621f06d22af41f32ad93bce46320979f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGuidedDecodingParams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request23getGuidedDecodingParamsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request17getAllottedTimeMsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request17getAllottedTimeMsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request17getAllottedTimeMsEv"></span><span id="tensorrt_llm::executor::Request::getAllottedTimeMsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a9cb91f2565d9973a5533b6f377673193"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16MillisecondsTypeE" title="tensorrt_llm::executor::MillisecondsType"><span class="n"><span class="pre">MillisecondsType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getAllottedTimeMs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request17getAllottedTimeMsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request12setStreamingEb">
<span id="_CPPv3N12tensorrt_llm8executor7Request12setStreamingEb"></span><span id="_CPPv2N12tensorrt_llm8executor7Request12setStreamingEb"></span><span id="tensorrt_llm::executor::Request::setStreaming__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae68a921ab34759ffc097e342253d52f9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setStreaming</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">streaming</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request12setStreamingEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig"></span><span id="tensorrt_llm::executor::Request::setSamplingConfig__SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1abe649c4845388cea78bf22c291538341"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSamplingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig"></span><span id="tensorrt_llm::executor::Request::setOutputConfig__OutputConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a73a7ec9973c9a76faf89738aedaa69f2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOutputConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request8setEndIdE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor7Request8setEndIdE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor7Request8setEndIdE10SizeType32"></span><span id="tensorrt_llm::executor::Request::setEndId__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a76a2cc1cc476cb0a050c24a73a6e5cbc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEndId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request8setEndIdE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request8setPadIdE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor7Request8setPadIdE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor7Request8setPadIdE10SizeType32"></span><span id="tensorrt_llm::executor::Request::setPadId__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a8f1d378a89c237f2c79f5510d285a32e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPadId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">padId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request8setPadIdE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request14setPositionIdsERKNSt6vectorI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor7Request14setPositionIdsERKNSt6vectorI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request14setPositionIdsERKNSt6vectorI10SizeType32EE"></span><span id="tensorrt_llm::executor::Request::setPositionIds__std::vector:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1afb5b1b10b3d2976d9f40320c63614122"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPositionIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">positionIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request14setPositionIdsERKNSt6vectorI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE"></span><span id="tensorrt_llm::executor::Request::setBadWords__std::list:VecTokens:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae2c911bc838c38ff9c0955063d532b5b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBadWords</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">badWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE"></span><span id="tensorrt_llm::executor::Request::setStopWords__std::list:VecTokens:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ac3fab641b66477d4c11446a7a09f9f72"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setStopWords</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stopWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor"></span><span id="tensorrt_llm::executor::Request::setEmbeddingBias__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a3184ce8fb36c13840dec0e219056e3e1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEmbeddingBias</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">embeddingBias</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig"></span><span id="tensorrt_llm::executor::Request::setExternalDraftTokensConfig__ExternalDraftTokensConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a311a94236786522ab77f47fe38bd0402"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">externalDraftTokensConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig"></span><span id="tensorrt_llm::executor::Request::setPromptTuningConfig__PromptTuningConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad01f7cf40b183ad8b7c77ee0c03d8c51"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPromptTuningConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">pTuningConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request14setMropeConfigERK11MropeConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request14setMropeConfigERK11MropeConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request14setMropeConfigERK11MropeConfig"></span><span id="tensorrt_llm::executor::Request::setMropeConfig__MropeConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae2a2b91a7a51f369b93965cb8ca4e479"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMropeConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="tensorrt_llm::executor::MropeConfig"><span class="n"><span class="pre">MropeConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">mRopeConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request14setMropeConfigERK11MropeConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig"></span><span id="tensorrt_llm::executor::Request::setLoraConfig__LoraConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ada0604e9f9989519d8595e5b0777d875"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLoraConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">loraConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request18setLookaheadConfigERK23LookaheadDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request18setLookaheadConfigERK23LookaheadDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request18setLookaheadConfigERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::Request::setLookaheadConfig__LookaheadDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aebd7ed087f0e1ebeb03edd9b44d3a28d"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLookaheadConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request18setLookaheadConfigERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request25setKvCacheRetentionConfigERK22KvCacheRetentionConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request25setKvCacheRetentionConfigERK22KvCacheRetentionConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request25setKvCacheRetentionConfigERK22KvCacheRetentionConfig"></span><span id="tensorrt_llm::executor::Request::setKvCacheRetentionConfig__KvCacheRetentionConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1acc2a684dd429a6e6a2f738b597ce3dad"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setKvCacheRetentionConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheRetentionConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request25setKvCacheRetentionConfigERK22KvCacheRetentionConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE"></span><span id="tensorrt_llm::executor::Request::setLogitsPostProcessorName__ssCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a26704f9863977008bd456b1aa49e3e55"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLogitsPostProcessorName</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logitsPostProcessorName</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens">
<span id="_CPPv3N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens"></span><span id="_CPPv2N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens"></span><span id="tensorrt_llm::executor::Request::setEncoderInputTokenIds__VecTokensCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1adf41e2b961d2bd01a43ea7ef17d19a01"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEncoderInputTokenIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderInputTokenIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request11setClientIdE6IdType">
<span id="_CPPv3N12tensorrt_llm8executor7Request11setClientIdE6IdType"></span><span id="_CPPv2N12tensorrt_llm8executor7Request11setClientIdE6IdType"></span><span id="tensorrt_llm::executor::Request::setClientId__IdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1abf1a51a1a1462745563462e323e725bd"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setClientId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">clientId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request11setClientIdE6IdType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request11setPriorityE12PriorityType">
<span id="_CPPv3N12tensorrt_llm8executor7Request11setPriorityE12PriorityType"></span><span id="_CPPv2N12tensorrt_llm8executor7Request11setPriorityE12PriorityType"></span><span id="tensorrt_llm::executor::Request::setPriority__PriorityType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1acfcea4db0cf194766c5fde1cf4ec5e77"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPriority</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12PriorityTypeE" title="tensorrt_llm::executor::PriorityType"><span class="n"><span class="pre">PriorityType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">priority</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request11setPriorityE12PriorityType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request27setReturnAllGeneratedTokensEb">
<span id="_CPPv3N12tensorrt_llm8executor7Request27setReturnAllGeneratedTokensEb"></span><span id="_CPPv2N12tensorrt_llm8executor7Request27setReturnAllGeneratedTokensEb"></span><span id="tensorrt_llm::executor::Request::setReturnAllGeneratedTokens__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a122fe4b9e64de4c5305663d1fc0711a6"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setReturnAllGeneratedTokens</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnAllGeneratedTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request27setReturnAllGeneratedTokensEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request14setRequestTypeERK11RequestType">
<span id="_CPPv3N12tensorrt_llm8executor7Request14setRequestTypeERK11RequestType"></span><span id="_CPPv2N12tensorrt_llm8executor7Request14setRequestTypeERK11RequestType"></span><span id="tensorrt_llm::executor::Request::setRequestType__RequestTypeCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1af880e937db1156c88361d1fe3b9e1ce1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRequestType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11RequestTypeE" title="tensorrt_llm::executor::RequestType"><span class="n"><span class="pre">RequestType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestType</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request14setRequestTypeERK11RequestType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request21setContextPhaseParamsE18ContextPhaseParams">
<span id="_CPPv3N12tensorrt_llm8executor7Request21setContextPhaseParamsE18ContextPhaseParams"></span><span id="_CPPv2N12tensorrt_llm8executor7Request21setContextPhaseParamsE18ContextPhaseParams"></span><span id="tensorrt_llm::executor::Request::setContextPhaseParams__ContextPhaseParams"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1adf0ae5e8364ea484c784c018a0521aab"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setContextPhaseParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">contextPhaseParams</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request21setContextPhaseParamsE18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputFeaturesE6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor7Request23setEncoderInputFeaturesE6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor7Request23setEncoderInputFeaturesE6Tensor"></span><span id="tensorrt_llm::executor::Request::setEncoderInputFeatures__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ac9b8af7d68183782ae71c49fa1d417ec"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEncoderInputFeatures</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">encoderInputFeatures</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputFeaturesE6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request22setEncoderOutputLengthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor7Request22setEncoderOutputLengthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor7Request22setEncoderOutputLengthE10SizeType32"></span><span id="tensorrt_llm::executor::Request::setEncoderOutputLength__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a4b485629537541897b7a5cac3b5a8da0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEncoderOutputLength</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">encoderOutputLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request22setEncoderOutputLengthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request21setCrossAttentionMaskE6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor7Request21setCrossAttentionMaskE6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor7Request21setCrossAttentionMaskE6Tensor"></span><span id="tensorrt_llm::executor::Request::setCrossAttentionMask__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1addc7a8f4af939975c638d46c25d8c04b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCrossAttentionMask</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">crossAttentionMask</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request21setCrossAttentionMaskE6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request21setNumReturnSequencesE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor7Request21setNumReturnSequencesE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor7Request21setNumReturnSequencesE10SizeType32"></span><span id="tensorrt_llm::executor::Request::setNumReturnSequences__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a9127fdc7bd7b2b40f3b79512f0de3aa0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNumReturnSequences</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numReturnSequences</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request21setNumReturnSequencesE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request14setEagleConfigERKNSt8optionalI11EagleConfigEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request14setEagleConfigERKNSt8optionalI11EagleConfigEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request14setEagleConfigERKNSt8optionalI11EagleConfigEE"></span><span id="tensorrt_llm::executor::Request::setEagleConfig__std::optional:EagleConfig:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1acb170f755a284ae5d474d6a23ef20faf"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEagleConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">eagleConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request14setEagleConfigERKNSt8optionalI11EagleConfigEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request22setSkipCrossAttnBlocksE6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor7Request22setSkipCrossAttnBlocksE6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor7Request22setSkipCrossAttnBlocksE6Tensor"></span><span id="tensorrt_llm::executor::Request::setSkipCrossAttnBlocks__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1af0d79c0e179a1e5274a75cc61a2f1fb2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSkipCrossAttnBlocks</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">skipCrossAttnBlocks</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request22setSkipCrossAttnBlocksE6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request23setGuidedDecodingParamsERK20GuidedDecodingParams">
<span id="_CPPv3N12tensorrt_llm8executor7Request23setGuidedDecodingParamsERK20GuidedDecodingParams"></span><span id="_CPPv2N12tensorrt_llm8executor7Request23setGuidedDecodingParamsERK20GuidedDecodingParams"></span><span id="tensorrt_llm::executor::Request::setGuidedDecodingParams__GuidedDecodingParamsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a0c0331051ef7d1a0f3a469c1671993b7"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setGuidedDecodingParams</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">guidedDecodingParams</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request23setGuidedDecodingParamsERK20GuidedDecodingParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request17setAllottedTimeMsE16MillisecondsType">
<span id="_CPPv3N12tensorrt_llm8executor7Request17setAllottedTimeMsE16MillisecondsType"></span><span id="_CPPv2N12tensorrt_llm8executor7Request17setAllottedTimeMsE16MillisecondsType"></span><span id="tensorrt_llm::executor::Request::setAllottedTimeMs__MillisecondsType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a384ac064c96a053a479d4213c35a1b48"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setAllottedTimeMs</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor16MillisecondsTypeE" title="tensorrt_llm::executor::MillisecondsType"><span class="n"><span class="pre">MillisecondsType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">allottedTimeMs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request17setAllottedTimeMsE16MillisecondsType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request16kDefaultPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor7Request16kDefaultPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request16kDefaultPriorityE"></span><span id="tensorrt_llm::executor::Request::kDefaultPriority__PriorityType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1af72bb695ce2e2024cb6dbf32e788a219"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12PriorityTypeE" title="tensorrt_llm::executor::PriorityType"><span class="n"><span class="pre">PriorityType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultPriority</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0.5</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request16kDefaultPriorityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE">
<span id="_CPPv3N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE"></span><span id="tensorrt_llm::executor::Request::kBatchedPostProcessorName__auto"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1add106668c254a78b6bc7432fcc08913e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBatchedPostProcessorName</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;batched&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE" title="Link to this definition"></a><br /></dt>
<dd><p>This logits postprocessor name will dispatch to the batched logits postprocessor. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor7Request5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request5mImplE"></span><span id="tensorrt_llm::executor::Request::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a2bef0e71b0eadc746b10462a419ff339"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseE">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseE"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseE"></span><span id="tensorrt_llm::executor::Response"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Class that holds either an error or a result. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringENSt8optionalI6IdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringENSt8optionalI6IdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringENSt8optionalI6IdTypeEE"></span><span id="tensorrt_llm::executor::Response::Response__IdType.ss.std::optional:IdType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1ac1c11ef972edc9ed1d4ee378b43592f7"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestId</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">errorMsg</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">clientId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringENSt8optionalI6IdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6ResultNSt8optionalI6IdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseE6IdType6ResultNSt8optionalI6IdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseE6IdType6ResultNSt8optionalI6IdTypeEE"></span><span id="tensorrt_llm::executor::Response::Response__IdType.Result.std::optional:IdType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a9c1ffcd57a905446841ab9379e514abe"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6ResultNSt8optionalI6IdTypeEE" title="tensorrt_llm::executor::Response::Response::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">Result</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">clientId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6ResultNSt8optionalI6IdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseD0Ev"></span><span id="tensorrt_llm::executor::Response::~Response"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1aef44ea215e23b7f97c69b3de2e2d612b"></span><span class="sig-name descname"><span class="n"><span class="pre">~Response</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseERK8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseERK8Response"></span><span id="tensorrt_llm::executor::Response::Response__ResponseCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1aa59cf06a2cd20bcbb414c6329f950653"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response" title="tensorrt_llm::executor::Response::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseERR8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseERR8Response"></span><span id="tensorrt_llm::executor::Response::Response__ResponseRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a6775c21a72390415490ec5138a4c91c3"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response" title="tensorrt_llm::executor::Response::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseaSERK8Response">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseaSERK8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseaSERK8Response"></span><span id="tensorrt_llm::executor::Response::assign-operator__ResponseCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1acda14277748b6b6f7538ff1f0636215d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseaSERK8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseaSERR8Response">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseaSERR8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseaSERR8Response"></span><span id="tensorrt_llm::executor::Response::assign-operator__ResponseRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a1292cbf4171fe10a8b2824daa6bb6ee4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseaSERR8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response12getRequestIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response12getRequestIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response12getRequestIdEv"></span><span id="tensorrt_llm::executor::Response::getRequestIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1adf694176a77659e4f4afcb4fd7112eea"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRequestId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response12getRequestIdEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the id of the request for which this response was generated. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response11getClientIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response11getClientIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response11getClientIdEv"></span><span id="tensorrt_llm::executor::Response::getClientIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a0ce0d259f14b7fc6a63db97e322bfc93"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getClientId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response11getClientIdEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the client id of the request for which this response was generated. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response8hasErrorEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response8hasErrorEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response8hasErrorEv"></span><span id="tensorrt_llm::executor::Response::hasErrorC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1acd0e0fe2d71ecd4e9a6ae7529d5a9958"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">hasError</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response8hasErrorEv" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if this response has an error or not. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response11getErrorMsgEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response11getErrorMsgEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response11getErrorMsgEv"></span><span id="tensorrt_llm::executor::Response::getErrorMsgC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a4eeac11a9ae40fbaeb28ef96451f5185"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getErrorMsg</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response11getErrorMsgEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the error msg for this response Will throw an exception if hasError is false. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response9getResultEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response9getResultEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response9getResultEv"></span><span id="tensorrt_llm::executor::Response::getResultC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a917149b976494cf2812c77cc87f16372"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">getResult</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response9getResultEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the result for this response Will throw an exception if hasResult is true. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor8Response5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor8Response5mImplE"></span><span id="tensorrt_llm::executor::Response::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a62ec31b2113d81890e913e6b5e4fddac"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6ResultE">
<span id="_CPPv3N12tensorrt_llm8executor6ResultE"></span><span id="_CPPv2N12tensorrt_llm8executor6ResultE"></span><span id="tensorrt_llm::executor::Result"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Result</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Struct that holds the generation result. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result7isFinalE">
<span id="_CPPv3N12tensorrt_llm8executor6Result7isFinalE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result7isFinalE"></span><span id="tensorrt_llm::executor::Result::isFinal__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1aa17894b95a897fc98e10059a95060097"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isFinal</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result7isFinalE" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if this is the final result for the request. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result14outputTokenIdsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result14outputTokenIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result14outputTokenIdsE"></span><span id="tensorrt_llm::executor::Result::outputTokenIds__BeamTokens"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a878f113efb1c2e087bece4205f61fbfb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="tensorrt_llm::executor::BeamTokens"><span class="n"><span class="pre">BeamTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">outputTokenIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result14outputTokenIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The output tokens for each beam. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result11cumLogProbsE"></span><span id="tensorrt_llm::executor::Result::cumLogProbs__std::optional:VecLogProbs:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1aa4463aefb92ec56587efff7b64ca3f97"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE" title="tensorrt_llm::executor::VecLogProbs"><span class="n"><span class="pre">VecLogProbs</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result11cumLogProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>The cumulative log probabilities. Size beamSize. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result8logProbsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result8logProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result8logProbsE"></span><span id="tensorrt_llm::executor::Result::logProbs__std::optional:std::vector:VecLogProbs::"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a11e7bc33f082ca62a83dbc12a734889c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE" title="tensorrt_llm::executor::VecLogProbs"><span class="n"><span class="pre">VecLogProbs</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result8logProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>The log probabilities for each generated token. Size [beamSize, outputLen]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result13contextLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result13contextLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result13contextLogitsE"></span><span id="tensorrt_llm::executor::Result::contextLogits__std::optional:Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1aade7284f001e6d0418709492b3b90cf1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result13contextLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>The context logits. Size [promptLen, vocabSizePadded]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result16generationLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result16generationLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result16generationLogitsE"></span><span id="tensorrt_llm::executor::Result::generationLogits__std::optional:Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a3c7678084cfb911c75b34ea2c4043d9b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generationLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result16generationLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>The generation logits. Size [beamSize, maxNewTokens, vocabSizePadded] (non-streaming) or [maxNewTokens, beamSize, vocabSizePadded] (streaming and allGeneratedTokens) or [1, beamSize, vocabSizePadded] (streaming and non-allGeneratedTokens) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result21specDecFastLogitsInfoE">
<span id="_CPPv3N12tensorrt_llm8executor6Result21specDecFastLogitsInfoE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result21specDecFastLogitsInfoE"></span><span id="tensorrt_llm::executor::Result::specDecFastLogitsInfo__std::optional:SpeculativeDecodingFastLogitsInfo:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a571a13ca4077accc2f47313125b34820"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE" title="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo"><span class="n"><span class="pre">SpeculativeDecodingFastLogitsInfo</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">specDecFastLogitsInfo</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result21specDecFastLogitsInfoE" title="Link to this definition"></a><br /></dt>
<dd><p>Logits information for direct transfer when using fast logits. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result13encoderOutputE">
<span id="_CPPv3N12tensorrt_llm8executor6Result13encoderOutputE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result13encoderOutputE"></span><span id="tensorrt_llm::executor::Result::encoderOutput__std::optional:Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1af1fe609c6a99ac08b1bf4c687ca8e654"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">encoderOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result13encoderOutputE" title="Link to this definition"></a><br /></dt>
<dd><p>The encoder output. Size [encoderLen, hiddenSize]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result13finishReasonsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result13finishReasonsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result13finishReasonsE"></span><span id="tensorrt_llm::executor::Result::finishReasons__std::vector:FinishReason:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a71979ade4af529c950547e687f2037a1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12FinishReasonE" title="tensorrt_llm::executor::FinishReason"><span class="n"><span class="pre">FinishReason</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">finishReasons</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result13finishReasonsE" title="Link to this definition"></a><br /></dt>
<dd><p>The reason why the model stopped generating tokens for each beam in this request. Size [beamSize]. Currently only supported when beamSize is 1 and when using BatchingType::kINFLIGHT. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result18contextPhaseParamsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result18contextPhaseParamsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result18contextPhaseParamsE"></span><span id="tensorrt_llm::executor::Result::contextPhaseParams__std::optional:ContextPhaseParams:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a8d585cd5ac9ffc418b4f7ed8b03b6590"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextPhaseParams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result18contextPhaseParamsE" title="Link to this definition"></a><br /></dt>
<dd><p>The params of the context phase. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result12decodingIterE">
<span id="_CPPv3N12tensorrt_llm8executor6Result12decodingIterE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result12decodingIterE"></span><span id="tensorrt_llm::executor::Result::decodingIter__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1ad0023b5973c19222f825d92099097469"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">decodingIter</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result12decodingIterE" title="Link to this definition"></a><br /></dt>
<dd><p>The number of the decoding iterations used to generate the result. In autoregressive decoding, it is equal to the maximum length of the beam in outputTokenIds. In speculative decoding, might be less than maximum length of the beam in outputTokenIds as more than one token can be generated per iteration. Used for speculative decoding statistics. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result13sequenceIndexE">
<span id="_CPPv3N12tensorrt_llm8executor6Result13sequenceIndexE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result13sequenceIndexE"></span><span id="tensorrt_llm::executor::Result::sequenceIndex__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a8e4211d45667d1632b40c9340f60e848"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">sequenceIndex</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result13sequenceIndexE" title="Link to this definition"></a><br /></dt>
<dd><p>The index of the output sequence of this result where 0 &lt;= sequenceIndex &lt; numReturnSequences. In beam search (beamWidth &gt; 1), this index will be always zero because all beams to be returned are included in this result. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result15isSequenceFinalE">
<span id="_CPPv3N12tensorrt_llm8executor6Result15isSequenceFinalE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result15isSequenceFinalE"></span><span id="tensorrt_llm::executor::Result::isSequenceFinal__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a477ccd08f3c10499af4c223ac0e71424"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isSequenceFinal</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result15isSequenceFinalE" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if this is the final result for a given sequence in the request In beam search (beamWidth &gt; 1), the value will always equal to the value of isFinal. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result18requestPerfMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result18requestPerfMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result18requestPerfMetricsE"></span><span id="tensorrt_llm::executor::Result::requestPerfMetrics__std::optional:RequestPerfMetrics:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a5108e30550019f4330573ae1b2a059fc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">requestPerfMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result18requestPerfMetricsE" title="Link to this definition"></a><br /></dt>
<dd><p>Performance metrics if returnPerfMetrics is set in <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1OutputConfig"><span class="std std-ref">OutputConfig</span></a>. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDurationE">
<span id="_CPPv3N12tensorrt_llm8executor28RetentionPriorityAndDurationE"></span><span id="_CPPv2N12tensorrt_llm8executor28RetentionPriorityAndDurationE"></span><span id="tensorrt_llm::executor::RetentionPriorityAndDuration"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RetentionPriorityAndDuration"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RetentionPriorityAndDuration</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDurationE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration28RetentionPriorityAndDurationERKNSt8optionalI17RetentionPriorityEERKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor28RetentionPriorityAndDuration28RetentionPriorityAndDurationERKNSt8optionalI17RetentionPriorityEERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor28RetentionPriorityAndDuration28RetentionPriorityAndDurationERKNSt8optionalI17RetentionPriorityEERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::RetentionPriorityAndDuration::RetentionPriorityAndDuration__std::optional:RetentionPriority:CR.std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RetentionPriorityAndDuration_1a97cd54dd5c5b97062851ab77062a1abb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RetentionPriorityAndDuration</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">retentionPriority</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">durationMs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration28RetentionPriorityAndDurationERKNSt8optionalI17RetentionPriorityEERKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration17retentionPriorityE">
<span id="_CPPv3N12tensorrt_llm8executor28RetentionPriorityAndDuration17retentionPriorityE"></span><span id="_CPPv2N12tensorrt_llm8executor28RetentionPriorityAndDuration17retentionPriorityE"></span><span id="tensorrt_llm::executor::RetentionPriorityAndDuration::retentionPriority__std::optional:RetentionPriority:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RetentionPriorityAndDuration_1a00d6d80567d8afbd37c114f12fda68b6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17RetentionPriorityE" title="tensorrt_llm::executor::RetentionPriority"><span class="n"><span class="pre">RetentionPriority</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">retentionPriority</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration17retentionPriorityE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration10durationMsE">
<span id="_CPPv3N12tensorrt_llm8executor28RetentionPriorityAndDuration10durationMsE"></span><span id="_CPPv2N12tensorrt_llm8executor28RetentionPriorityAndDuration10durationMsE"></span><span id="tensorrt_llm::executor::RetentionPriorityAndDuration::durationMs__std::optional:std::chrono::milliseconds:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RetentionPriorityAndDuration_1ad1906adfd260cd18f9e6855309cf4f0f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">durationMs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor28RetentionPriorityAndDuration10durationMsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfigE"></span><span id="tensorrt_llm::executor::SamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Sampling configuration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::SamplingConfig__SizeType32.std::optional:SizeType32:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:TokenIdType:CR.std::optional:FloatType:CR.std::optional:RandomSeedType:CR.std::optional:FloatType:CR.std::optional:SizeType32:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:SizeType32:CR.std::optional:SizeType32:CR.std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a7e97bbb1a0430eec62b80fe197c0cc5f"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topK</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topP</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPMin</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPResetIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPDecay</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">seed</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">temperature</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">beamSearchDiversityRate</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">repetitionPenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">presencePenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">frequencyPenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lengthPenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">earlyStopping</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">numReturnSequences</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd><p>Constructor for <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1SamplingConfig"><span class="std std-ref">SamplingConfig</span></a> See description of parameters below. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig"></span><span id="tensorrt_llm::executor::SamplingConfig::eq-operator__SamplingConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a08919f8eb49492f6e0f23f20b2ff0555"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getBeamWidthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9f3845e5dabe1f14aa2130c7adc8dc9d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBeamWidth</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig17getNumReturnBeamsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig17getNumReturnBeamsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig17getNumReturnBeamsEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getNumReturnBeamsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab12e1b657d4a05e2ab49c51d19671f28"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumReturnBeams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig17getNumReturnBeamsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopKEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig7getTopKEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig7getTopKEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopKC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a2946f6278862e85ef92d085376f12cbf"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopK</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopKEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig7getTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig7getTopPEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a71d2f03b643676951fa7729d0804edd1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPMinC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1afd69a9dcf61940b09f83819c8a3bf849"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopPMin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPResetIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5b077cc8420200b24a127e9bb0f84cf1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopPResetIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPDecayC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ae495fcf888bb96f6b252e60ea42be9a4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopPDecay</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getSeedEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig7getSeedEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig7getSeedEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getSeedC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a404891b17704f007b37acfcab3bd6738"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSeed</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getSeedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getRandomSeedC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ada61e8e574d1b77c5a20f40128a6b062"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRandomSeed</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTemperatureC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a8ac079c1742f930190ed0e488a0e9368"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTemperature</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getMinTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getMinTokensEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getMinTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a0c0860e9f2280c3a35fdcd6e523af493"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMinTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getMinLengthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a016868c14c93204a411d2a5271687d15"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMinLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getBeamSearchDiversityRateC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1afdca37090c7283d286973e12af8d6c8b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBeamSearchDiversityRate</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getRepetitionPenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a4a39513b2dbeccdd039cec455a1916e3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getPresencePenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af2ba08c30a1fceaa151bc3ce4e04dd19"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPresencePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getFrequencyPenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a106a21dae3dff8379ef1792a56fe46a6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getLengthPenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a44fd8a1cc4a45d757aac154362118fbf"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLengthPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getEarlyStoppingC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ae8b64e9aecc63bf976ec9ae55e20f162"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEarlyStopping</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getNoRepeatNgramSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac3e9205236940fa410ba06df4e0ed4a8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig21getNumReturnSequencesEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig21getNumReturnSequencesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig21getNumReturnSequencesEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getNumReturnSequencesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a8db12d54d1b1521a96b0c193051556f8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumReturnSequences</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig21getNumReturnSequencesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32"></span><span id="tensorrt_llm::executor::SamplingConfig::setBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9d408d845ffd468e5c77a12644580acc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBeamWidth</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopK__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5461168ba842d21d25c4c584d9f9d023"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopK</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topK</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopP__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a913f54f64881543acd7709ed99da5085"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopP</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topP</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopPMin__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aa1789d40088cb15c5a07398607f2dd76"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopPMin</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPMin</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopPResetIds__std::optional:TokenIdType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab4926172b96ae86559d6247ba0b55b51"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopPResetIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPResetIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopPDecay__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab897fc960092c5e35a20e5eee4849d2d"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopPDecay</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPDecay</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig7setSeedERKNSt8optionalI14RandomSeedTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig7setSeedERKNSt8optionalI14RandomSeedTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig7setSeedERKNSt8optionalI14RandomSeedTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setSeed__std::optional:RandomSeedType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a28cf7200764ca1e5769e7d1ac4ca936c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSeed</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">seed</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setSeedERKNSt8optionalI14RandomSeedTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setRandomSeed__std::optional:RandomSeedType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a7cc025262a443afb49884619e7d0dbd4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRandomSeed</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">randomSeed</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTemperature__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a22735fef3e4a93fb9d6f05e2405c8fc0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTemperature</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">temperature</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinTokensERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setMinTokensERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setMinTokensERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setMinTokens__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ae64136223dada66024961ce6e689a069"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMinTokens</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinTokensERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setMinLength__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a8cd60f4e1ba6a48483ae02608eb31385"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMinLength</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setBeamSearchDiversityRate__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab22c4602e2c4ad41a8b5896d914d9a0c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBeamSearchDiversityRate</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">beamSearchDiversityRate</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setRepetitionPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac58bfcdc98186d18963554bb4b6c01f8"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">repetitionPenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setPresencePenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1abe8dd822d08b0210e9f1fa3b36490fdf"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPresencePenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">presencePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setFrequencyPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a468c9e296264e676bd40c884c85bc4da"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">frequencyPenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setLengthPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9ea9013d86101dcdfb9d15a2b25998f5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLengthPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lengthPenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setEarlyStopping__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a20097f9517e3f46e6584addf31a8fb3f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEarlyStopping</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">earlyStopping</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setNoRepeatNgramSize__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1add1f574a466f0c5a89933fe6f0d3dc13"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig21setNumReturnSequencesERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig21setNumReturnSequencesERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig21setNumReturnSequencesERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setNumReturnSequences__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af3e76091583d194b540be92db684ea76"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNumReturnSequences</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">numReturnSequences</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig21setNumReturnSequencesERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig20updateNumReturnBeamsEv">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig20updateNumReturnBeamsEv"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig20updateNumReturnBeamsEv"></span><span id="tensorrt_llm::executor::SamplingConfig::updateNumReturnBeams"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af7693a504a1486108435bcb2cd8e523e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">updateNumReturnBeams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20updateNumReturnBeamsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE"></span><span id="tensorrt_llm::executor::SamplingConfig::mBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab75403888fa3c8d301912dbc350c35fb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBeamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE" title="Link to this definition"></a><br /></dt>
<dd><p>The beam width. Default is 1 which disables beam search. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopKE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig5mTopKE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig5mTopKE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopK__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac9850671f2d8eed17ecd6e8e4845f401"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopK</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopKE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls number of logits to sample from. Default is 0 (all logits). </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopPE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig5mTopPE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig5mTopPE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopP__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac442a2bcecf64be99729c882c9aaf09b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopP</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopPE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the top-P probability to sample from. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig8mTopPMinE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig8mTopPMinE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig8mTopPMinE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopPMin__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac0692f29ae762728c4a7d1e438dad91e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopPMin</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig8mTopPMinE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls decay in the top-P algorithm. topPMin is lower-bound. Default is 1.e-6. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopPResetIds__std::optional:TokenIdType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab0c410d2557fdf327ede1d67c292a7a2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopPResetIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls decay in the top-P algorithm. Indicates where to reset the decay. Default is 1. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopPDecay__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af161a37e4aa10723d8ae5627943b4e03"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopPDecay</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls decay in the top-P algorithm. The decay value. Default is 1.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig5mSeedE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig5mSeedE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig5mSeedE"></span><span id="tensorrt_llm::executor::SamplingConfig::mSeed__std::optional:RandomSeedType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a3146e1b95e12aea945d14e0365bfe642"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSeed</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mSeedE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the random seed used by the random number generator in sampling. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12mTemperatureE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12mTemperatureE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12mTemperatureE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTemperature__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab5b9a3e150bdc536de54725c9210281e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTemperature</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12mTemperatureE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the modulation of logits when sampling new tokens. It can have values &gt; 0.f. Default is 1.0f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10mMinTokensE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10mMinTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10mMinTokensE"></span><span id="tensorrt_llm::executor::SamplingConfig::mMinTokens__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a1bd2df7e1d0d9e90e26126ab0e6f7632"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMinTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mMinTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Lower bound on the number of tokens to generate. Values &lt; 1 have no effect. Default is 1. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE"></span><span id="tensorrt_llm::executor::SamplingConfig::mBeamSearchDiversityRate__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5d7fb8264ede8714d68ee323af7ce58f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBeamSearchDiversityRate</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the diversity in beam search. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mRepetitionPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1afc6184e856da9b06c54fd9093ca517bc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRepetitionPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Used to penalize tokens based on how often they appear in the sequence. It can have any value &gt; 0.f. Values &lt; 1.f encourages repetition, values &gt; 1.f discourages it. Default is 1.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mPresencePenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1acee024920425ee593d16ac95113d5e2d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPresencePenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Used to penalize tokens already present in the sequence (irrespective of the number of appearances). It can have any values. Values &lt; 0.f encourage repetition, values &gt; 0.f discourage it. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mFrequencyPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a537a6f894e3c3093957c46774b23bb38"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFrequencyPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Used to penalize tokens already present in the sequence (dependent on the number of appearances). It can have any values. Values &lt; 0.f encourage repetition, values &gt; 0.f discourage it. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mLengthPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a131e7f83f8b2a827761284f1bca2f934"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLengthPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls how to penalize longer sequences in beam search. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE"></span><span id="tensorrt_llm::executor::SamplingConfig::mEarlyStopping__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5bf3c3f8361b9ffd284f386ccd69eab0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEarlyStopping</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls whether the generation process finishes once beamWidth sentences are generated (ends with end_token) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE"></span><span id="tensorrt_llm::executor::SamplingConfig::mNoRepeatNgramSize__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a782d671e6803a1cd7c3116004082b42e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNoRepeatNgramSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls how many repeat ngram size are acceptable. Default is 1 &lt;&lt; 30. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig19mNumReturnSequencesE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig19mNumReturnSequencesE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig19mNumReturnSequencesE"></span><span id="tensorrt_llm::executor::SamplingConfig::mNumReturnSequences__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aa36b13555a137de54e1a58106405e527"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumReturnSequences</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig19mNumReturnSequencesE" title="Link to this definition"></a><br /></dt>
<dd><p>The number of return sequences or beams. In beam search, the value should be less than or equal to mBeamWidth. In sampling, it specifies the total number of independently generated sequences. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig15mNumReturnBeamsE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig15mNumReturnBeamsE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig15mNumReturnBeamsE"></span><span id="tensorrt_llm::executor::SamplingConfig::mNumReturnBeams__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aa9fd268711326300faebb1d64fc7aab0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumReturnBeams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig15mNumReturnBeamsE" title="Link to this definition"></a><br /></dt>
<dd><p>The number of beams to return. It is equal to beamWidth unless numReturnSequences is set. If beamWidth &gt; 1 and numReturnSequences is set, then numReturnBeams is equal to numReturnSequences. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32"></span><span id="tensorrt_llm::executor::SamplingConfig::checkBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a766b90d2d0f211808369d11906e561f6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">checkBeamWidth</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopK__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac12283a0962b4087301acdb60aa7fef7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopK</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topK</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopP__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aa113c8eab8bf08c0824d6d836778ced7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopP</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topP</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopPMin__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a82e79c31f7329090750a7612848a8a58"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopPMin</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPMin</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopPResetIds__std::optional:TokenIdType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1adf7a7cbc22eb3017045c09c3afebf54c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopPResetIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPResetIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopPDecay__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a113ee3f518a17e0057edc1bdf6271c20"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopPDecay</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPDecay</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTemperature__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a182335063c59582c15b7c5b0a74db6cc"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTemperature</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">temperature</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkRepetitionPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a3ade3b38da1bfc92dc350f1342bd96ab"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">penalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkMinTokensERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14checkMinTokensERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14checkMinTokensERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkMinTokens__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af59789692b81d08bbdc9c905bc061efe"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkMinTokens</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkMinTokensERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkNoRepeatNgramSize__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9c0bf40fa2438f1bdbd2f815916c65bf"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkBeamSearchDiversityRate__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a49233a524c5e20983c880ca94f724e85"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkBeamSearchDiversityRate</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">beamSearchDiversityRate</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig23checkNumReturnSequencesERKNSt8optionalI10SizeType32EE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig23checkNumReturnSequencesERKNSt8optionalI10SizeType32EE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig23checkNumReturnSequencesERKNSt8optionalI10SizeType32EE10SizeType32"></span><span id="tensorrt_llm::executor::SamplingConfig::checkNumReturnSequences__std::optional:SizeType32:CR.SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a8ab5975ec921e4bf93a9a244509ca56a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkNumReturnSequences</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">numReturnSequences</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig23checkNumReturnSequencesERKNSt8optionalI10SizeType32EE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfigE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfigE"></span><span id="tensorrt_llm::executor::SchedulerConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SchedulerConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the scheduler. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEENSt8optionalI18DynamicBatchConfigEE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEENSt8optionalI18DynamicBatchConfigEE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEENSt8optionalI18DynamicBatchConfigEE"></span><span id="tensorrt_llm::executor::SchedulerConfig::SchedulerConfig__CapacitySchedulerPolicy.std::optional:ContextChunkingPolicy:.std::optional:DynamicBatchConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1abe564b1c96406b37274f08781d30b1d1"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SchedulerConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">capacitySchedulerPolicy</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE" title="tensorrt_llm::executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT"><span class="n"><span class="pre">kGUARANTEED_NO_EVICT</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">contextChunkingPolicy</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="tensorrt_llm::executor::DynamicBatchConfig"><span class="n"><span class="pre">DynamicBatchConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dynamicBatchConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEENSt8optionalI18DynamicBatchConfigEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15SchedulerConfigeqERK15SchedulerConfig">
<span id="_CPPv3NK12tensorrt_llm8executor15SchedulerConfigeqERK15SchedulerConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor15SchedulerConfigeqERK15SchedulerConfig"></span><span id="tensorrt_llm::executor::SchedulerConfig::eq-operator__SchedulerConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1aa13c9b6ed3407575091225a06481b1a6"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfigeqERK15SchedulerConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv">
<span id="_CPPv3NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv"></span><span id="tensorrt_llm::executor::SchedulerConfig::getCapacitySchedulerPolicyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1aa72f57d07d885b6fd256189911076b8b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCapacitySchedulerPolicy</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv">
<span id="_CPPv3NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv"></span><span id="tensorrt_llm::executor::SchedulerConfig::getContextChunkingPolicyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1a4e32082e39a6be5a295a0b334b2956d0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getContextChunkingPolicy</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15SchedulerConfig21getDynamicBatchConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor15SchedulerConfig21getDynamicBatchConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15SchedulerConfig21getDynamicBatchConfigEv"></span><span id="tensorrt_llm::executor::SchedulerConfig::getDynamicBatchConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1a88b50045152cc75c4c036a989bded5bf"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="tensorrt_llm::executor::DynamicBatchConfig"><span class="n"><span class="pre">DynamicBatchConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDynamicBatchConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig21getDynamicBatchConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE"></span><span id="tensorrt_llm::executor::SchedulerConfig::mCapacitySchedulerPolicy__CapacitySchedulerPolicy"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1a3c5dc1e328e369821e9b510bb89ddd0f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCapacitySchedulerPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p>The capacity scheduler policy. See CapacitySchedulerPolicy. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE"></span><span id="tensorrt_llm::executor::SchedulerConfig::mContextChunkingPolicy__std::optional:ContextChunkingPolicy:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1aa3f547ef1160624ae6e9ddce782967cf"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mContextChunkingPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p>The context chunking policy. See ContextChunkingPolicy. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig19mDynamicBatchConfigE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig19mDynamicBatchConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig19mDynamicBatchConfigE"></span><span id="tensorrt_llm::executor::SchedulerConfig::mDynamicBatchConfig__std::optional:DynamicBatchConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1aaf3ec00bf67afe2ff3f0a7bb8268b6d8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="tensorrt_llm::executor::DynamicBatchConfig"><span class="n"><span class="pre">DynamicBatchConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDynamicBatchConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig19mDynamicBatchConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The config for tuning batch size dynamically. See DynamicBatchSizeConfig. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor25SpeculativeDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor25SpeculativeDecodingConfigE"></span><span id="tensorrt_llm::executor::SpeculativeDecodingConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SpeculativeDecodingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for speculative decoding (both draft and target models) </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfig25SpeculativeDecodingConfigEb">
<span id="_CPPv3N12tensorrt_llm8executor25SpeculativeDecodingConfig25SpeculativeDecodingConfigEb"></span><span id="_CPPv2N12tensorrt_llm8executor25SpeculativeDecodingConfig25SpeculativeDecodingConfigEb"></span><span id="tensorrt_llm::executor::SpeculativeDecodingConfig::SpeculativeDecodingConfig__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SpeculativeDecodingConfig_1ac32ec832b367409979415f6e1bd7d7bb"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">fastLogits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfig25SpeculativeDecodingConfigEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25SpeculativeDecodingConfigeqERK25SpeculativeDecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor25SpeculativeDecodingConfigeqERK25SpeculativeDecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor25SpeculativeDecodingConfigeqERK25SpeculativeDecodingConfig"></span><span id="tensorrt_llm::executor::SpeculativeDecodingConfig::eq-operator__SpeculativeDecodingConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SpeculativeDecodingConfig_1a1dab7496894ab51899cb8b809579eba0"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25SpeculativeDecodingConfigeqERK25SpeculativeDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfig10fastLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor25SpeculativeDecodingConfig10fastLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor25SpeculativeDecodingConfig10fastLogitsE"></span><span id="tensorrt_llm::executor::SpeculativeDecodingConfig::fastLogits__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SpeculativeDecodingConfig_1a669e6c71682aba0dcb1fb12856604b4d"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">fastLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfig10fastLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>Send logits tensor directly from draft to target model. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE">
<span id="_CPPv3N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE"></span><span id="_CPPv2N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE"></span><span id="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1SpeculativeDecodingFastLogitsInfo"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SpeculativeDecodingFastLogitsInfo</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Struct that holds the logits information when using direct transfer. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo8toTensorEv">
<span id="_CPPv3NK12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo8toTensorEv"></span><span id="_CPPv2NK12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo8toTensorEv"></span><span id="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo::toTensorC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1SpeculativeDecodingFastLogitsInfo_1ae4477369ea3c05b9e41148f94419fbde"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toTensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo8toTensorEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the struct serialized into a tensor that can be used as generation logits input. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo14draftRequestIdE">
<span id="_CPPv3N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo14draftRequestIdE"></span><span id="_CPPv2N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo14draftRequestIdE"></span><span id="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo::draftRequestId__uint64_t"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1SpeculativeDecodingFastLogitsInfo_1ace8ba8a3ce4cf4586a7c48a95eeaf377"></span><span class="n"><span class="pre">uint64_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">draftRequestId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo14draftRequestIdE" title="Link to this definition"></a><br /></dt>
<dd><p>Draft request id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo18draftParticipantIdE">
<span id="_CPPv3N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo18draftParticipantIdE"></span><span id="_CPPv2N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo18draftParticipantIdE"></span><span id="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo::draftParticipantId__int32_t"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1SpeculativeDecodingFastLogitsInfo_1a032fad3fc1dafa3f30ac89efad1fb484"></span><span class="n"><span class="pre">int32_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">draftParticipantId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfo18draftParticipantIdE" title="Link to this definition"></a><br /></dt>
<dd><p>MPI world rank of the draft model leader. </p>
</dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm3mpiE">
<span id="_CPPv3N12tensorrt_llm3mpiE"></span><span id="_CPPv2N12tensorrt_llm3mpiE"></span><span id="tensorrt_llm::mpi"></span><span class="target" id="namespacetensorrt__llm_1_1mpi"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mpi</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm3mpiE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="serialization-h">
<h2>serialization.h<a class="headerlink" href="#serialization-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13SerializationE">
<span id="_CPPv3N12tensorrt_llm8executor13SerializationE"></span><span id="_CPPv2N12tensorrt_llm8executor13SerializationE"></span><span id="tensorrt_llm::executor::Serialization"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Serialization</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13SerializationE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization20deserializeTimePointERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization20deserializeTimePointERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization20deserializeTimePointERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeTimePoint__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2d477b7cbaa81ccdce8b0228da633407"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeTimePoint</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeTimePointERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN18RequestPerfMetrics9TimePointERNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKN18RequestPerfMetrics9TimePointERNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKN18RequestPerfMetrics9TimePointERNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestPerfMetrics::TimePointCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ada4597a81d9f13a07a92924a0d887444"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tp</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN18RequestPerfMetrics9TimePointERNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN18RequestPerfMetrics9TimePointE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERKN18RequestPerfMetrics9TimePointE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERKN18RequestPerfMetrics9TimePointE"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestPerfMetrics::TimePointCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a47205a5c23884ff19818607562d9565c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN18RequestPerfMetrics9TimePointE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializeRequestPerfMetricsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializeRequestPerfMetricsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializeRequestPerfMetricsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequestPerfMetrics__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a75fbc151569f640c8867ccea7f154284"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequestPerfMetrics</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeRequestPerfMetricsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18RequestPerfMetricsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18RequestPerfMetricsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18RequestPerfMetricsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestPerfMetricsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a7780e617369c0cafdc7218ab69455499"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">metrics</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18RequestPerfMetricsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18RequestPerfMetrics">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18RequestPerfMetrics"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18RequestPerfMetrics"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestPerfMetricsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9727f8f81dbc73c2f57c06ac18392b91"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics"><span class="n"><span class="pre">RequestPerfMetrics</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">metrics</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18RequestPerfMetrics" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSamplingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a018e2c20ed62e05f0428c770990cf3a7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSamplingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__SamplingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab250085b9f35d5c2ca33e63241f4ffa5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2b0d1cd68b238760ff02f8a4740bead3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeOutputConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1afa9afb4d8f345960ce3419aa50a7aecb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeOutputConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__OutputConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a50563b0a86ded8ca3f7273d126ac7042"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__OutputConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aa51e9d72a4d69d15f2371d2eb8cbeba3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeExternalDraftTokensConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2bca807ceb86f58a385de9bdcc1bf481"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ExternalDraftTokensConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1adbcd97e4e4d2822a1222fab34c3b3699"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ExternalDraftTokensConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a0c631d80dfe19e428b64e92e24ecbfc7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializePromptTuningConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9c20dc9bf0ec815a2fd91243e79f82a1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializePromptTuningConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__PromptTuningConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aeffa6dbca81617e4c8f2f151402de0aa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__PromptTuningConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a64114e901f6976ad2ede341a4ce46623"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization22deserializeMropeConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization22deserializeMropeConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization22deserializeMropeConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeMropeConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab8a512a4577e8df9c91b4fab9bed3ed5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="tensorrt_llm::executor::MropeConfig"><span class="n"><span class="pre">MropeConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeMropeConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeMropeConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11MropeConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK11MropeConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK11MropeConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__MropeConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aec3a044c9bd6672dd486f6881843c67c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="tensorrt_llm::executor::MropeConfig"><span class="n"><span class="pre">MropeConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11MropeConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11MropeConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK11MropeConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK11MropeConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__MropeConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a887668fed866430fd21e2d0ea93f2c07"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11MropeConfigE" title="tensorrt_llm::executor::MropeConfig"><span class="n"><span class="pre">MropeConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11MropeConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeLoraConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab6ae74552a93ff397c5af265c344fe56"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeLoraConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__LoraConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aab0cf47956bc7bb1ee1452aa90edb6de"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__LoraConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a70a62ef2cdbc5a65dc3b61e4052a8133"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization20deserializeCommStateERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization20deserializeCommStateERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization20deserializeCommStateERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeCommState__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a34443e07959170ea25fde6ef27452c97"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CommState</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeCommState</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeCommStateERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache9CommStateERNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache9CommStateERNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache9CommStateERNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__kv_cache::CommStateCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a10777655cc16bcc53cd25de031bf04cd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CommState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache9CommStateERNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache9CommStateE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache9CommStateE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache9CommStateE"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__kv_cache::CommStateCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae3050aad60f9b26b95e5359353596359"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CommState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache9CommStateE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization22deserializeSocketStateERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization22deserializeSocketStateERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization22deserializeSocketStateERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSocketState__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2428a248a3d2e88853dca9f8fbd60d5a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">SocketState</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSocketState</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeSocketStateERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache11SocketStateERNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache11SocketStateERNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache11SocketStateERNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__kv_cache::SocketStateCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8e412592cb9710e2d7a37df8a5decc9f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">SocketState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache11SocketStateERNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache11SocketStateE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache11SocketStateE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache11SocketStateE"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__kv_cache::SocketStateCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab85a5c8fd7ec4c2ff14cb51b738b71d1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">SocketState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache11SocketStateE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization21deserializeCacheStateERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization21deserializeCacheStateERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization21deserializeCacheStateERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeCacheState__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a0703b669e35401e746cfa9a4ebe63ae2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CacheState</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeCacheState</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization21deserializeCacheStateERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache10CacheStateERNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache10CacheStateERNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache10CacheStateERNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__kv_cache::CacheStateCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1af1d1dfa6808bf6e306cfb816b1021f4c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CacheState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN8kv_cache10CacheStateERNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache10CacheStateE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache10CacheStateE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache10CacheStateE"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__kv_cache::CacheStateCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aa2b26e9bdbcdc241a96a864ca2e6905a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="tensorrt_llm::executor::kv_cache"><span class="n"><span class="pre">kv_cache</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">CacheState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN8kv_cache10CacheStateE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization31deserializeDataTransceiverStateERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization31deserializeDataTransceiverStateERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization31deserializeDataTransceiverStateERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDataTransceiverState__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ac72aa5b4757d07a6178867c6c706e897"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">DataTransceiverState</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDataTransceiverState</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization31deserializeDataTransceiverStateERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20DataTransceiverStateRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK20DataTransceiverStateRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK20DataTransceiverStateRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DataTransceiverStateCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a15e862f6a3af7233b8b1171273421b8e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">DataTransceiverState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dataTransceiverState</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20DataTransceiverStateRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20DataTransceiverState">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK20DataTransceiverState"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK20DataTransceiverState"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DataTransceiverStateCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6ca4f00139b3b74fe3638e9e0f4e33e1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">DataTransceiverState</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dataTransceiverState</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20DataTransceiverState" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializeContextPhaseParamsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializeContextPhaseParamsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializeContextPhaseParamsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeContextPhaseParams__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a94714fbc91d588c87fc573c0f08c710e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeContextPhaseParams</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeContextPhaseParamsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18ContextPhaseParamsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18ContextPhaseParamsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18ContextPhaseParamsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ContextPhaseParamsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aff42ad9117d0656ec2f7a1a9d30e5a16"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">contextPhaseParams</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18ContextPhaseParamsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18ContextPhaseParams">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18ContextPhaseParams"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18ContextPhaseParams"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ContextPhaseParamsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a31e8e60215fe54fc7ae438da792ffb7b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18ContextPhaseParamsE" title="tensorrt_llm::executor::ContextPhaseParams"><span class="n"><span class="pre">ContextPhaseParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">contextPhaseParams</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18ContextPhaseParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequest__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a52ea6afb73b932c514d8bf2fbacb5df2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequest</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a51b1f66893f945e1d48ce4f466ba1010"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a15092605072969164a22559fcf61f6c1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeTensor__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab32ab17b2c60fba0ee29097c2ea7e055"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeTensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__TensorCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8fe83cad52a85278ba6ff00c542a9214"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a85dbca4cdebadd45d7329329fcf656c0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization32deserializeSpecDecFastLogitsInfoERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization32deserializeSpecDecFastLogitsInfoERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization32deserializeSpecDecFastLogitsInfoERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSpecDecFastLogitsInfo__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aad3b65ee245884aa7b8984bd688be641"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE" title="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo"><span class="n"><span class="pre">SpeculativeDecodingFastLogitsInfo</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSpecDecFastLogitsInfo</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization32deserializeSpecDecFastLogitsInfoERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK33SpeculativeDecodingFastLogitsInfoRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK33SpeculativeDecodingFastLogitsInfoRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK33SpeculativeDecodingFastLogitsInfoRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__SpeculativeDecodingFastLogitsInfoCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a973afba71c86101e4105c9c10f625714"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE" title="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo"><span class="n"><span class="pre">SpeculativeDecodingFastLogitsInfo</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">info</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK33SpeculativeDecodingFastLogitsInfoRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK33SpeculativeDecodingFastLogitsInfo">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK33SpeculativeDecodingFastLogitsInfo"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK33SpeculativeDecodingFastLogitsInfo"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__SpeculativeDecodingFastLogitsInfoCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a27e38a765aa5bd176bbbbfdd50489627"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33SpeculativeDecodingFastLogitsInfoE" title="tensorrt_llm::executor::SpeculativeDecodingFastLogitsInfo"><span class="n"><span class="pre">SpeculativeDecodingFastLogitsInfo</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">info</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK33SpeculativeDecodingFastLogitsInfo" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeResult__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a258edbaa27d4bc82e5919f921aaff5b3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeResult</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ResultCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae2bc066744d82a2457a974478a92d24d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">result</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ResultCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1acf908575acc37c7e106488e59f8aa4ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">result</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeResponse__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae5a8b5e299fb2231fc41881b2fd12b31"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeResponse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ResponseCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a1dc5a101175bb9de92efa650e6d20e68"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">response</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ResponseCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1accb642acb602af27140cce717b5b18e8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">response</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeResponses__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a14d75bf69426bfa81113b5efe6d4a5cd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE"></span><span id="tensorrt_llm::executor::Serialization::serialize__std::vector:Response:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ac95ee049bd2b4fc2d401e0596739df3d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">responses</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeKvCacheConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a93df88f47d0cbfa681af463ed1b680d8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeKvCacheConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__KvCacheConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8e27bfce7f4b56448cdb1e40596ad5b6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__KvCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad457e1d524480c49cc90ac55aeda1943"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializeDynamicBatchConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializeDynamicBatchConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializeDynamicBatchConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDynamicBatchConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ade33854bf404f5410198d2b54e33f6c0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="tensorrt_llm::executor::DynamicBatchConfig"><span class="n"><span class="pre">DynamicBatchConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDynamicBatchConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeDynamicBatchConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18DynamicBatchConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18DynamicBatchConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18DynamicBatchConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DynamicBatchConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab8c488d557304327ad8704a319fb807a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="tensorrt_llm::executor::DynamicBatchConfig"><span class="n"><span class="pre">DynamicBatchConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dynamicBatchConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18DynamicBatchConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18DynamicBatchConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18DynamicBatchConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18DynamicBatchConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DynamicBatchConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad0dbc1675cda40c8096a0356791f4c19"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18DynamicBatchConfigE" title="tensorrt_llm::executor::DynamicBatchConfig"><span class="n"><span class="pre">DynamicBatchConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">dynamicBatchConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18DynamicBatchConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSchedulerConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5721cef6b72c2a079b5dbe94a7f9249e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSchedulerConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__SchedulerConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a75d2e0606543f60ed2bf6bbaf00456f4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__SchedulerConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1af3642582eaeb939ba265089c2fee753a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization40deserializeExtendedRuntimePerfKnobConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization40deserializeExtendedRuntimePerfKnobConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization40deserializeExtendedRuntimePerfKnobConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeExtendedRuntimePerfKnobConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ac507a4af186242ca004a666769d156d1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeExtendedRuntimePerfKnobConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization40deserializeExtendedRuntimePerfKnobConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK29ExtendedRuntimePerfKnobConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK29ExtendedRuntimePerfKnobConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK29ExtendedRuntimePerfKnobConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ExtendedRuntimePerfKnobConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5b53e64d28f954ae520fba4d4395b070"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">extendedRuntimePerfKnobConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK29ExtendedRuntimePerfKnobConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK29ExtendedRuntimePerfKnobConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK29ExtendedRuntimePerfKnobConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK29ExtendedRuntimePerfKnobConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ExtendedRuntimePerfKnobConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9f2fe7ee62a55bf3a2b033e4a9556674"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor29ExtendedRuntimePerfKnobConfigE" title="tensorrt_llm::executor::ExtendedRuntimePerfKnobConfig"><span class="n"><span class="pre">ExtendedRuntimePerfKnobConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">extendedRuntimePerfKnobConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK29ExtendedRuntimePerfKnobConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeParallelConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8ddf1b2798dbf98d3eac0759c4646360"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeParallelConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ParallelConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a352018daa6065ab5d7f8d40df594f790"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">parallelConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ParallelConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a903b5bc4b669a8e8cfdff9206512969a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">parallelConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializePeftCacheConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a36ab38bb0f5fe55b9829c8177c93e91f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializePeftCacheConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__PeftCacheConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a28c920c3cbb2c7f97678a0ed2fd704cb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__PeftCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae6b46926ad9c8754e7cd89c1b9c8bd36"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeOrchestratorConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae025cd58097d96dc74cb1acd6207325d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeOrchestratorConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__OrchestratorConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a7864913d0c2362e972654c2a5613b566"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__OrchestratorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a255cb93a98ccdf748db37f8b492520d3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDecodingMode__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6793f998581467d3540e2ed35167be15"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDecodingMode</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DecodingModeCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a0b3f4dd50c3174e5522cd945a55416a8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingMode</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DecodingModeCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a68100e2eb882d29677a4c9ed45cfd7a8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingMode</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeLookaheadDecodingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a09e3d04d4394b505c41a2f88d2395de2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeLookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__LookaheadDecodingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a7f7d5716bc415282ad7d870cee8a0af2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__LookaheadDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5dce85091047f4b569a106630ddd0840"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization22deserializeEagleConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization22deserializeEagleConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization22deserializeEagleConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeEagleConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1abfe63c04a67358325f7de9c1a84bd5d9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeEagleConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeEagleConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11EagleConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK11EagleConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK11EagleConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__EagleConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a37588c9152027aec9b1470c1aab5cca7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">eagleConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11EagleConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11EagleConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK11EagleConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK11EagleConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__EagleConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a0c47859bf5f406af776bca0e1dc91a83"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11EagleConfigE" title="tensorrt_llm::executor::EagleConfig"><span class="n"><span class="pre">EagleConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">eagleConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11EagleConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization36deserializeSpeculativeDecodingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization36deserializeSpeculativeDecodingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization36deserializeSpeculativeDecodingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSpeculativeDecodingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9715efdef8cfea9f207849268600fa00"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSpeculativeDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeSpeculativeDecodingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25SpeculativeDecodingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK25SpeculativeDecodingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK25SpeculativeDecodingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__SpeculativeDecodingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae22a12c73325733708048218506f7ee9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">specDecConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25SpeculativeDecodingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25SpeculativeDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK25SpeculativeDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK25SpeculativeDecodingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__SpeculativeDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9c03fac4af9f93b327425530008079ae"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25SpeculativeDecodingConfigE" title="tensorrt_llm::executor::SpeculativeDecodingConfig"><span class="n"><span class="pre">SpeculativeDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">specDecConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25SpeculativeDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeGuidedDecodingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8a1d553784a718951697122d7f0f83a5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeGuidedDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__GuidedDecodingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a070aaf03c054306d09b32a63a72e22ca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">guidedDecodingConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__GuidedDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6df9f55c33fc9bfbb0237293824af7ca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingConfigE" title="tensorrt_llm::executor::GuidedDecodingConfig"><span class="n"><span class="pre">GuidedDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">guidedDecodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingParamsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingParamsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingParamsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeGuidedDecodingParams__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1afa984172377b9e1e3c65fcf107178d44"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeGuidedDecodingParams</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization31deserializeGuidedDecodingParamsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingParamsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingParamsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingParamsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__GuidedDecodingParamsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1abaa14972d252807ca10179e9d836c3dc"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">guidedDecodingParams</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK20GuidedDecodingParamsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingParams">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingParams"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingParams"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__GuidedDecodingParamsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a48b0a0db79fd6aa7cc5dda8f4cc91dc4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor20GuidedDecodingParamsE" title="tensorrt_llm::executor::GuidedDecodingParams"><span class="n"><span class="pre">GuidedDecodingParams</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">guidedDecodingParams</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK20GuidedDecodingParams" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization33deserializeKvCacheRetentionConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization33deserializeKvCacheRetentionConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization33deserializeKvCacheRetentionConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeKvCacheRetentionConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae10d1c31c6521f41ec81003c1a186baa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeKvCacheRetentionConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization33deserializeKvCacheRetentionConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK22KvCacheRetentionConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK22KvCacheRetentionConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK22KvCacheRetentionConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__KvCacheRetentionConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2ce73676e73612b6c129a1c22550ef91"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheRetentionConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK22KvCacheRetentionConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK22KvCacheRetentionConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK22KvCacheRetentionConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK22KvCacheRetentionConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__KvCacheRetentionConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a26cbc2d54e6ba3c1e68c52e6943810f1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheRetentionConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK22KvCacheRetentionConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization36deserializeTokenRangeRetentionConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization36deserializeTokenRangeRetentionConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization36deserializeTokenRangeRetentionConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeTokenRangeRetentionConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a915e5c3142c7fc92365b4d58bef33596"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeTokenRangeRetentionConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeTokenRangeRetentionConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigERNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigERNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigERNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__KvCacheRetentionConfig::TokenRangeRetentionConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2e2917d1129a25aa6e6772c13c0e269a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tokenRangeRetentionConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigERNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigE"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__KvCacheRetentionConfig::TokenRangeRetentionConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aa5e510b891a5e3b45e5f6d0cb2af3176"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig"><span class="n"><span class="pre">KvCacheRetentionConfig</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="tensorrt_llm::executor::KvCacheRetentionConfig::TokenRangeRetentionConfig"><span class="n"><span class="pre">TokenRangeRetentionConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tokenRangeRetentionConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERKN22KvCacheRetentionConfig25TokenRangeRetentionConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDecodingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a60e192f644c0e8693f0a3b12d6bb60e0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DecodingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad1a7a61b52c5d823406a5a2bb05edb21"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8b8d2707b981fbb788a24a31c1170a72"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization22deserializeDebugConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization22deserializeDebugConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization22deserializeDebugConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDebugConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9308aeea31b9888f81b340b19772bf10"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDebugConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization22deserializeDebugConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11DebugConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK11DebugConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK11DebugConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DebugConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1afa49754c743c041ae7c7cdace53bed38"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">debugConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK11DebugConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11DebugConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK11DebugConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK11DebugConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DebugConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aa75fc302341a6adb991d635ef4e2ba0b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11DebugConfigE" title="tensorrt_llm::executor::DebugConfig"><span class="n"><span class="pre">DebugConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">debugConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK11DebugConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeExecutorConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a3bdc71d1f61ffbe8192eec4c69a6f863"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeExecutorConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ExecutorConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8d327c320f459d69fd6561a420558674"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aeabfd8c6625ad85aed6ef2cb72f8cd66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeKvCacheStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a00364ba63856dbd99f89ae1e45770ffa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeKvCacheStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__KvCacheStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a92003eb85d2979660e29056b4f81cf0a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__KvCacheStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5e3c1533994cd82ce444b0c632bb6fbe"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeStaticBatchingStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad66b899cda69588f655b08dd36f15925"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeStaticBatchingStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__StaticBatchingStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a1187b7a88a0e5deb38ff8e593372e984"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">staticBatchingStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__StaticBatchingStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6fe6208a4f78db241e6d9c6b2afd24e5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">staticBatchingStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeInflightBatchingStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6ddddf159c364fc1af5d7ba7a1a5c9cf"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeInflightBatchingStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__InflightBatchingStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a1e571817bc0585695894442bd4697fc6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inflightBatchingStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__InflightBatchingStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1adfd862d5d685d90dfbec4e742c340cd7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inflightBatchingStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeIterationStats__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9797c0d1af10c396b36f548de7d2e8e2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeIterationStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeIterationStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad1bbfdc68d6c6d0a8342317fac32bd93"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeIterationStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__IterationStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2ed4d91cf95dfd30083fa0154f8d6243"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats"></span><span id="tensorrt_llm::executor::Serialization::serialize__IterationStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6950808f968c435d4efa2696421e49be"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__IterationStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a24a30c0a00c745499a73d7f754c9e67a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI14IterationStatsEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI14IterationStatsEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI14IterationStatsEE"></span><span id="tensorrt_llm::executor::Serialization::serialize__std::vector:IterationStats:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a18501fffc9db30dbb1def7281d639d04"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStatsVec</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI14IterationStatsEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization28deserializeIterationStatsVecERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization28deserializeIterationStatsVecERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization28deserializeIterationStatsVecERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeIterationStatsVec__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad25c05a8bcff2152ffae00bcfd3c6553"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeIterationStatsVec</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization28deserializeIterationStatsVecERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization33deserializeDisServingRequestStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization33deserializeDisServingRequestStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization33deserializeDisServingRequestStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDisServingRequestStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae3f88de690b770067d41e9de565365da"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE" title="tensorrt_llm::executor::DisServingRequestStats"><span class="n"><span class="pre">DisServingRequestStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDisServingRequestStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization33deserializeDisServingRequestStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK22DisServingRequestStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK22DisServingRequestStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK22DisServingRequestStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DisServingRequestStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a4b68bd59f29204220da5cf065f98fc10"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE" title="tensorrt_llm::executor::DisServingRequestStats"><span class="n"><span class="pre">DisServingRequestStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK22DisServingRequestStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK22DisServingRequestStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK22DisServingRequestStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK22DisServingRequestStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DisServingRequestStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad55ef9f5fa3d7225411143f51d3e7297"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE" title="tensorrt_llm::executor::DisServingRequestStats"><span class="n"><span class="pre">DisServingRequestStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">disServingRequestStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK22DisServingRequestStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeRequestStageERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeRequestStageERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeRequestStageERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequestStage__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1abb6dd3b8c12d6d3a9921e39749bbdcda"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="tensorrt_llm::executor::RequestStage"><span class="n"><span class="pre">RequestStage</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequestStage</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeRequestStageERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12RequestStageRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12RequestStageRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12RequestStageRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestStageCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a696465e544a9f4a6377d17ab029d47dc"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="tensorrt_llm::executor::RequestStage"><span class="n"><span class="pre">RequestStage</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStage</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12RequestStageRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStage">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStage"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStage"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestStageCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1acdd5dd282522b9bb0ff940bd2c7c1d86"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="tensorrt_llm::executor::RequestStage"><span class="n"><span class="pre">RequestStage</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStage</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStage" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeRequestStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeRequestStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeRequestStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequestStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a36d0b08bddac8c6c9252effde1127b86"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequestStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeRequestStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12RequestStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12RequestStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12RequestStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a64b2978d2361e3d5c31a72f82ea99c30"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12RequestStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aed0abb450ac08a383e844ff44fb721d9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12RequestStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequestStatsPerIteration__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a44549f08640743ae8609e2d9aea5f8aa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequestStatsPerIteration</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequestStatsPerIteration__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a490b8d2bef2ca33876374bf40ea54588"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequestStatsPerIteration</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization35deserializeRequestStatsPerIterationERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIterationRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIterationRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIterationRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestStatsPerIterationCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a76bea696802f2256018e9ae72c30c9a4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIterationRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIteration">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIteration"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIteration"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestStatsPerIterationCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ac6b4c8f3e213096649b7c76d85911231"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK24RequestStatsPerIteration" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK24RequestStatsPerIteration">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK24RequestStatsPerIteration"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK24RequestStatsPerIteration"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestStatsPerIterationCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5ab0e83eb6c825653558f13aa3730480"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK24RequestStatsPerIteration" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI24RequestStatsPerIterationEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI24RequestStatsPerIterationEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI24RequestStatsPerIterationEE"></span><span id="tensorrt_llm::executor::Serialization::serialize__std::vector:RequestStatsPerIteration:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5ff3fdcfdc4c9c9cca1716c0efdbf04b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStatsVec</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI24RequestStatsPerIterationEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization38deserializeRequestStatsPerIterationVecERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization38deserializeRequestStatsPerIterationVecERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization38deserializeRequestStatsPerIterationVecERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequestStatsPerIterationVec__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a215b502eda1023303c092284649e5e0b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequestStatsPerIterationVec</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization38deserializeRequestStatsPerIterationVecERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeString__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ac520ac5f4cc9ec6f5b6bf53ab5b7da8f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeString</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeBool__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a19d98b5105004255595bfeecd7cecb4a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeBool</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeModelType__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9944e16a2c2b64e441b36fa72afb2e6b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeModelType</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8kv_cacheE">
<span id="_CPPv3N12tensorrt_llm8executor8kv_cacheE"></span><span id="_CPPv2N12tensorrt_llm8executor8kv_cacheE"></span><span id="tensorrt_llm::executor::kv_cache"></span><span class="target" id="namespacetensorrt__llm_1_1executor_1_1kv__cache"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kv_cache</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8kv_cacheE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="tensor-h">
<h2>tensor.h<a class="headerlink" href="#tensor-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5ShapeE">
<span id="_CPPv3N12tensorrt_llm8executor5ShapeE"></span><span id="_CPPv2N12tensorrt_llm8executor5ShapeE"></span><span id="tensorrt_llm::executor::Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ArrayView</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detailE" title="tensorrt_llm::executor::detail"><span class="n"><span class="pre">detail</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E" title="tensorrt_llm::executor::detail::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape4BaseE">
<span id="_CPPv3N12tensorrt_llm8executor5Shape4BaseE"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape4BaseE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1abea4be68b98825ce00bd278dd61492ac"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Base</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ArrayView</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detailE" title="tensorrt_llm::executor::detail"><span class="n"><span class="pre">detail</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E" title="tensorrt_llm::executor::detail::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape9DimType64E">
<span id="_CPPv3N12tensorrt_llm8executor5Shape9DimType64E"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape9DimType64E"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1a83c46d18b03c349f865a290209be5528"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DimType64</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_cv_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE" title="tensorrt_llm::executor::Shape::Base"><span class="n"><span class="pre">Base</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">value_type</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape5ShapeEv">
<span id="_CPPv3N12tensorrt_llm8executor5Shape5ShapeEv"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape5ShapeEv"></span><span id="tensorrt_llm::executor::Shape::Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1a476b2b6a441e9fcbb8cad03b354b6681"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE">
<span id="_CPPv3N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE"></span><span id="tensorrt_llm::executor::Shape::Shape__DimType64CP.Base::size_type"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1aadbc8651b94baa75a3e2eb2cff889293"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E" title="tensorrt_llm::executor::Shape::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE" title="tensorrt_llm::executor::Shape::Base"><span class="n"><span class="pre">Base</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_type</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE">
<span id="_CPPv3N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE"></span><span id="tensorrt_llm::executor::Shape::Shape__std::initializer_list:DimType64:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1a9211bd067fe9f04c94a49decf4b812de"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">initializer_list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E" title="tensorrt_llm::executor::Shape::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensorE">
<span id="_CPPv3N12tensorrt_llm8executor6TensorE"></span><span id="_CPPv2N12tensorrt_llm8executor6TensorE"></span><span id="tensorrt_llm::executor::Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab192218140e150441c3809208318577d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToCpu__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a50aa2db7bc31318e493941918a304f6a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToCpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToPinned__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a97a17d0f122678e4269d578f8bbcc057"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToPooledPinned__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a772e702ed0364b8fe922b1488e06f42d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToPooledPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToManaged__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a82d22baf219b3cf39e0ad7ea010e5457"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToManaged</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToGpu__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1aab06656d158bf52a97fdb554273bedd7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToGpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorEv">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorEv"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorEv"></span><span id="tensorrt_llm::executor::Tensor::Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ae79dbd3ecc8694d366f5e2b67556a8cd"></span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensorD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor6TensorD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor6TensorD0Ev"></span><span id="tensorrt_llm::executor::Tensor::~Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a28b3397651c85c4065e25d8d10d1c103"></span><span class="sig-name descname"><span class="n"><span class="pre">~Tensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::Tensor__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a785597fe1ac8b56286d4ad836ec70035"></span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor" title="tensorrt_llm::executor::Tensor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorERR6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorERR6Tensor"></span><span id="tensorrt_llm::executor::Tensor::Tensor__TensorRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ae40ad8a4bf115014f6ed52d674b07461"></span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor" title="tensorrt_llm::executor::Tensor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensoraSERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6TensoraSERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6TensoraSERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::assign-operator__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a9a84b33d4e8f6106873b4cc03439534b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensoraSERR6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6TensoraSERR6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6TensoraSERR6Tensor"></span><span id="tensorrt_llm::executor::Tensor::assign-operator__TensorRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ad02487337036149be898932409f1f696"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERR6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7getDataEv">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="tensorrt_llm::executor::Tensor::getData"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a420f7872623102fb87eab00b8df4a0d9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getData</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7getDataEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor7getDataEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="tensorrt_llm::executor::Tensor::getDataC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab4e6ad75a64407f0fb8caa362098e3ab"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getData</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getDataEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor11getDataTypeEv"></span><span id="tensorrt_llm::executor::Tensor::getDataTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a745614bf656424a3576690a25b5afbd4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor11getDataTypeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the data type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv"></span><span id="tensorrt_llm::executor::Tensor::getMemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab60d4a2939254f259d6354714bb3396a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryTypeE" title="tensorrt_llm::executor::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMemoryType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the memory type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor8getShapeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor8getShapeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor8getShapeEv"></span><span id="tensorrt_llm::executor::Tensor::getShapeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab6f07c9a28710562e4366187b6d465cb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getShape</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor8getShapeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the tensor dimensions. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor7getSizeEv"></span><span id="tensorrt_llm::executor::Tensor::getSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a65ca052e56fa4b4b819456678a80cb82"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getSizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the number of elements in the tensor. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv"></span><span id="tensorrt_llm::executor::Tensor::getSizeInBytesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ae05d5b4735c1aeb2536903d51c2dc2fd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSizeInBytes</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the size of the tensor in bytes. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr"></span><span id="tensorrt_llm::executor::Tensor::setZero__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ad0cfa186ced1eb92039987e745aa5174"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setZero</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Set the entire memory to zero. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>stream</strong> – Must be a valid CUDA stream if the memory type is GPU. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr"></span><span id="tensorrt_llm::executor::Tensor::setFrom__TensorCR.CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab74daacf178945a17ecd8552ad6ea733"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Copy the data and shape from another tensor. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>other</strong> – A tensor to copy from. </p></li>
<li><p><strong>stream</strong> – Must be a valid CUDA stream if the memory type is GPU. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6TensorcvbEv">
<span id="_CPPv3NK12tensorrt_llm8executor6TensorcvbEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6TensorcvbEv"></span><span id="tensorrt_llm::executor::Tensor::castto-b-operatorC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a6c8f0a832643e66193ac9246423769ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6TensorcvbEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6TensoreqERK6Tensor">
<span id="_CPPv3NK12tensorrt_llm8executor6TensoreqERK6Tensor"></span><span id="_CPPv2NK12tensorrt_llm8executor6TensoreqERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::eq-operator__TensorCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a80ff07f5b580df282b96a573c8e5bd5a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">rhs</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6TensoreqERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6TensorneERK6Tensor">
<span id="_CPPv3NK12tensorrt_llm8executor6TensorneERK6Tensor"></span><span id="_CPPv2NK12tensorrt_llm8executor6TensorneERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::neq-operator__TensorCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a35337e466b30dab925a4e895406214ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">!=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">rhs</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6TensorneERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::cpu__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ac116935be582b663662c6f39c7b774a5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a cpu tensor with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor3cpuE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor3cpuE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor3cpuE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a7ddcb397b6da797ca7825e93841abf19"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3cpuE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::pinned__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1aced97f5afb664a7dfc416a02b587462a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a cpu tensor in pinned memory with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor6pinnedE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor6pinnedE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor6pinnedE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ada1a99697b667dd46221f73a9a9ee7b9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor6pinnedE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::pooledPinned__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a7ba670c87ea79d7e66d562c22d7821ca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pooledPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a cpu tensor in pooled pinned memory with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a66ac66e737e5481c8f1640f15ce613fc"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pooledPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::managed__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a7731d566daed52054eccabfe3de81e1a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">managed</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a tensor in managed memory (UVM) with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor7managedE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor7managedE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor7managedE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1aa95d4b6ccfcc20b47fb6a0145073c57d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">managed</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor7managedE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape"></span><span id="tensorrt_llm::executor::Tensor::gpu__DataType.CudaStreamPtr.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a3fe1404ac1edb89baebaeb852817823d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a gpu tensor with the given shape and data type on a particular cuda stream.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor3gpuE6Tensor13CudaStreamPtr5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor3gpuE13CudaStreamPtr5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor3gpuE13CudaStreamPtr5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a0bbc0df61a4c8c3187278b66f38f41fe"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3gpuE6Tensor13CudaStreamPtr5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape"></span><span id="tensorrt_llm::executor::Tensor::of__DataType.voidP.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a53fce3c66408ad1da79dc838ea9a1921"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">of</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Wrap a data pointer into a tensor without taking ownership.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor2ofEP1T5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor2ofEP1T5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1abb953f082c26af69c559128e3e30b8ca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">of</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape" title="tensorrt_llm::executor::Tensor::of::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Wrap a data pointer into a tensor without taking ownership.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor2ofER1T"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor2ofER1T"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a367caf7b52fc7ccef5fe8e879376743b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">of</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T" title="tensorrt_llm::executor::Tensor::of::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">data</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T" title="Link to this definition"></a><br /></dt>
<dd><p>Wrap any container into a tensor without taking ownership.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor4ImplE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor4ImplE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor4ImplE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a499dd2cff3f5f1ebd0a2a508462af729"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Impl</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="tensorrt_llm::executor::Tensor::Tensor__std::shared_ptr:runtime::ITensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1abaff3fb79d1193b805d9ed9182f9482f"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr"></span><span id="tensorrt_llm::executor::Tensor::copyTo__std::shared_ptr:Impl:.CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1adce5c2600b80e6aac03d72f62c442eaa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyTo</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE" title="tensorrt_llm::executor::Tensor::Impl"><span class="n"><span class="pre">Impl</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7mTensorE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7mTensorE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7mTensorE"></span><span id="tensorrt_llm::executor::Tensor::mTensor__std::shared_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1af5858fa2abb457b83fc8aa0cfa0392ce"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE" title="tensorrt_llm::executor::Tensor::Impl"><span class="n"><span class="pre">Impl</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTensor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7mTensorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeE8DataTypev">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeEv"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeEv"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a323f2561238fa6191bf4b18b4916a464"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRuntimeType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeE8DataTypev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::detail::toITensor__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a6cf491a789adc04fcedfc1022a5167dd"></span><span class="k"><span class="pre">friend</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">toITensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="tensorrt_llm::executor::Tensor::detail::ofITensor__std::shared_ptr:runtime::ITensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a8e986804106e3fe17218fbeebcb2659c"></span><span class="k"><span class="pre">friend</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ofITensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detailE">
<span id="_CPPv3N12tensorrt_llm8executor6detailE"></span><span id="_CPPv2N12tensorrt_llm8executor6detailE"></span><span id="tensorrt_llm::executor::detail"></span><span class="target" id="namespacetensorrt__llm_1_1executor_1_1detail"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">detail</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detailE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detail9DimType64E">
<span id="_CPPv3N12tensorrt_llm8executor6detail9DimType64E"></span><span id="_CPPv2N12tensorrt_llm8executor6detail9DimType64E"></span><span class="target" id="tensor_8h_1aaa88c014789556bcf4f890ef4f91842b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DimType64</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">int64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detail9toITensorERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6detail9toITensorERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6detail9toITensorERK6Tensor"></span><span id="tensorrt_llm::executor::detail::toITensor__TensorCR"></span><span class="target" id="tensor_8h_1a688a8726508dd792068204eb5cf8273b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">toITensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detail9toITensorERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE">
<span id="_CPPv3N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="_CPPv2N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="tensorrt_llm::executor::detail::ofITensor__std::shared_ptr:runtime::ITensor:"></span><span class="target" id="tensor_8h_1acb29416de14060f4546f8653af9220e8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ofITensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimeE">
<span id="_CPPv3N12tensorrt_llm7runtimeE"></span><span id="_CPPv2N12tensorrt_llm7runtimeE"></span><span id="tensorrt_llm::runtime"></span><span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="types-h">
<h2>types.h<a class="headerlink" href="#types-h" title="Link to this heading"></a></h2>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::int8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4_1aaed76141f49476650f91cfbec3367e13"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E" title="tensorrt_llm::executor::DataType::kINT8"><span class="n"><span class="pre">kINT8</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::int32_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4_1a506e37fcee8102b90f320257e12ec485"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E" title="tensorrt_llm::executor::DataType::kINT32"><span class="n"><span class="pre">kINT32</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::int64_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4_1ae1e174a531a0abf5c31ed44525da252d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="tensorrt_llm::executor::DataType::kINT64"><span class="n"><span class="pre">kINT64</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::uint8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4_1a36e64efaf2f35f9e9d97c62fb4e74f84"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E" title="tensorrt_llm::executor::DataType::kUINT8"><span class="n"><span class="pre">kUINT8</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm8executor9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm8executor9TensorPtrE"></span><span class="target" id="types_8h_1a32a3846eb7d506ec2f4699f052f54dda"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10SizeType32E">
<span id="_CPPv3N12tensorrt_llm8executor10SizeType32E"></span><span id="_CPPv2N12tensorrt_llm8executor10SizeType32E"></span><span class="target" id="types_8h_1ad818c2e487265ea3ec0ddd760b768085"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType32</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9FloatTypeE">
<span id="_CPPv3N12tensorrt_llm8executor9FloatTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor9FloatTypeE"></span><span class="target" id="types_8h_1a48053cc72a5a67b3c19c817fb963ecea"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FloatType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11TokenIdTypeE">
<span id="_CPPv3N12tensorrt_llm8executor11TokenIdTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor11TokenIdTypeE"></span><span class="target" id="types_8h_1a5658d78655723ba6fac682b945cc6788"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenIdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9VecTokensE">
<span id="_CPPv3N12tensorrt_llm8executor9VecTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor9VecTokensE"></span><span class="target" id="types_8h_1a3f6e6a65b5e81747e30820e27d184e91"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">VecTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10BeamTokensE">
<span id="_CPPv3N12tensorrt_llm8executor10BeamTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor10BeamTokensE"></span><span class="target" id="types_8h_1afc3b526e44121eda1f3344e1d611f688"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BeamTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6IdTypeE">
<span id="_CPPv3N12tensorrt_llm8executor6IdTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor6IdTypeE"></span><span class="target" id="types_8h_1ab9563a6f39b5785365973b84532f9353"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor16VecTokenExtraIdsE">
<span id="_CPPv3N12tensorrt_llm8executor16VecTokenExtraIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor16VecTokenExtraIdsE"></span><span class="target" id="types_8h_1a3075281db711a71a376ec24fe11dd2c7"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">VecTokenExtraIds</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor16VecTokenExtraIdsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13IterationTypeE">
<span id="_CPPv3N12tensorrt_llm8executor13IterationTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor13IterationTypeE"></span><span class="target" id="types_8h_1ad5297846b0ec6db8536e14e970e0e09c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IterationType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE">
<span id="_CPPv3N12tensorrt_llm8executor14RandomSeedTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor14RandomSeedTypeE"></span><span class="target" id="types_8h_1aa7776f1267bf68fcfd4228fb0cc38bfc"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RandomSeedType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11VecLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor11VecLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor11VecLogProbsE"></span><span class="target" id="types_8h_1af438a899c644096dfd518bee78fad2dd"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">VecLogProbs</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9StreamPtrE">
<span id="_CPPv3N12tensorrt_llm8executor9StreamPtrE"></span><span id="_CPPv2N12tensorrt_llm8executor9StreamPtrE"></span><span class="target" id="types_8h_1a37eb662d6bef7e3702dac46671c3f543"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor16MillisecondsTypeE">
<span id="_CPPv3N12tensorrt_llm8executor16MillisecondsTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor16MillisecondsTypeE"></span><span class="target" id="types_8h_1a45b73aeba0f7c11070630d46720bfe32"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MillisecondsType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor16MillisecondsTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE">
<span id="_CPPv3N12tensorrt_llm8executor19LogitsPostProcessorE"></span><span id="_CPPv2N12tensorrt_llm8executor19LogitsPostProcessorE"></span><span class="target" id="types_8h_1a2ec193a9bb683333faee35fb0a801320"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessor</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="tensorrt_llm::executor::BeamTokens"><span class="n"><span class="pre">BeamTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE" title="tensorrt_llm::executor::StreamPtr"><span class="n"><span class="pre">StreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE">
<span id="_CPPv3N12tensorrt_llm8executor22LogitsPostProcessorMapE"></span><span id="_CPPv2N12tensorrt_llm8executor22LogitsPostProcessorMapE"></span><span class="target" id="types_8h_1ad609221eb06a044bdfa9afc31ac4b69b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unordered_map</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE" title="tensorrt_llm::executor::LogitsPostProcessor"><span class="n"><span class="pre">LogitsPostProcessor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE">
<span id="_CPPv3N12tensorrt_llm8executor26LogitsPostProcessorBatchedE"></span><span id="_CPPv2N12tensorrt_llm8executor26LogitsPostProcessorBatchedE"></span><span class="target" id="types_8h_1a0a7b5ffd7a5228373ede89d57bf68236"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">reference_wrapper</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="tensorrt_llm::executor::BeamTokens"><span class="n"><span class="pre">BeamTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE" title="tensorrt_llm::executor::StreamPtr"><span class="n"><span class="pre">StreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13MedusaChoicesE">
<span id="_CPPv3N12tensorrt_llm8executor13MedusaChoicesE"></span><span id="_CPPv2N12tensorrt_llm8executor13MedusaChoicesE"></span><span class="target" id="types_8h_1a6ee0e4eac427f8b44cd02cad361df400"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MedusaChoices</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12EagleChoicesE">
<span id="_CPPv3N12tensorrt_llm8executor12EagleChoicesE"></span><span id="_CPPv2N12tensorrt_llm8executor12EagleChoicesE"></span><span class="target" id="types_8h_1a79d9befd7352757e61700637c1a2fe84"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">EagleChoices</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12EagleChoicesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12PriorityTypeE">
<span id="_CPPv3N12tensorrt_llm8executor12PriorityTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor12PriorityTypeE"></span><span class="target" id="types_8h_1a80b67583441a85102ace282b7e2f72af"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PriorityType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12PriorityTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10BufferViewE">
<span id="_CPPv3N12tensorrt_llm8executor10BufferViewE"></span><span id="_CPPv2N12tensorrt_llm8executor10BufferViewE"></span><span class="target" id="types_8h_1a4f7ffe88ce0e39d4b8e29be9384975c7"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BufferView</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">basic_string_view</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10BufferViewE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-enums">Enums</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataTypeE">
<span id="_CPPv3N12tensorrt_llm8executor8DataTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor8DataTypeE"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96e"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kBOOLE"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kBOOLE"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea65d294ade613161e5ea557e935133213"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBOOL</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType6kUINT8E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType6kUINT8E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea159244e0ab6cea87daa7eaa8f6e19937"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUINT8</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kINT8E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kINT8E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kINT8E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea69c1a4a69db0e50820cf63122f90ad09"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINT8</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType6kINT32E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType6kINT32E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType6kINT32E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96eabd073fcbb15020b25a70e2cd95f9f4a9"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINT32</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType6kINT64E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType6kINT64E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType6kINT64E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea223b86006bb063f2a58200621e0656d5"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINT64</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kBF16E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kBF16E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kBF16E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea7b2caf2ed7b5f4f177fe3207cd198791"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBF16</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBF16E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType4kFP8E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType4kFP8E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType4kFP8E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea7397615c6bee5b62289fc7cceb82fbf7"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFP8</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType4kFP8E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kFP16E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kFP16E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kFP16E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea56e4ef5e47a48568bd24c4e0aaabcead"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFP16</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP16E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kFP32E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kFP32E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kFP32E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea6c6463a7e81555667de0d7b49101a701"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFP32</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP32E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType8kUNKNOWNE">
<span id="_CPPv3N12tensorrt_llm8executor8DataType8kUNKNOWNE"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType8kUNKNOWNE"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea16bb40b9102367393ecf8213078d1c6e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUNKNOWN</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType8kUNKNOWNE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11RequestTypeE">
<span id="_CPPv3N12tensorrt_llm8executor11RequestTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor11RequestTypeE"></span><span class="target" id="types_8h_1a2fa0faddbb57562b0cb8008594af7e39"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11RequestTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11RequestType35REQUEST_TYPE_CONTEXT_AND_GENERATIONE">
<span id="_CPPv3N12tensorrt_llm8executor11RequestType35REQUEST_TYPE_CONTEXT_AND_GENERATIONE"></span><span id="_CPPv2N12tensorrt_llm8executor11RequestType35REQUEST_TYPE_CONTEXT_AND_GENERATIONE"></span><span class="target" id="types_8h_1a2fa0faddbb57562b0cb8008594af7e39a99e4eb3c524741c99350cd470c463547"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">REQUEST_TYPE_CONTEXT_AND_GENERATION</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11RequestType35REQUEST_TYPE_CONTEXT_AND_GENERATIONE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11RequestType25REQUEST_TYPE_CONTEXT_ONLYE">
<span id="_CPPv3N12tensorrt_llm8executor11RequestType25REQUEST_TYPE_CONTEXT_ONLYE"></span><span id="_CPPv2N12tensorrt_llm8executor11RequestType25REQUEST_TYPE_CONTEXT_ONLYE"></span><span class="target" id="types_8h_1a2fa0faddbb57562b0cb8008594af7e39a2f1a58e3c83a5e91847f15370f6493e2"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">REQUEST_TYPE_CONTEXT_ONLY</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11RequestType25REQUEST_TYPE_CONTEXT_ONLYE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11RequestType28REQUEST_TYPE_GENERATION_ONLYE">
<span id="_CPPv3N12tensorrt_llm8executor11RequestType28REQUEST_TYPE_GENERATION_ONLYE"></span><span id="_CPPv2N12tensorrt_llm8executor11RequestType28REQUEST_TYPE_GENERATION_ONLYE"></span><span class="target" id="types_8h_1a2fa0faddbb57562b0cb8008594af7e39a71b2d6a7108f89c26847d4c49a1ac0a1"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">REQUEST_TYPE_GENERATION_ONLY</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11RequestType28REQUEST_TYPE_GENERATION_ONLYE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryTypeE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryTypeE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aab"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType4kCPUE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType4kCPUE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType4kCPUE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba2ce310327f474afc9f6774faa2f57903"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kCPUE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aabaa12edb33c24c693b60e591681cfb1e66"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU_PINNED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType15kCPU_PINNEDPOOLE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType15kCPU_PINNEDPOOLE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType15kCPU_PINNEDPOOLE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aabaf1e403dae17a973d2e1558c16e1cd1ef"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU_PINNEDPOOL</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType15kCPU_PINNEDPOOLE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType4kGPUE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType4kGPUE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType4kGPUE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba37ae524b76587efa776affdc5cdf2ac1"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kGPUE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType4kUVME">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType4kUVME"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType4kUVME"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba7d7de0bd70e1276255098b25010bdeb6"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUVM</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kUVME" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType8kUNKNOWNE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType8kUNKNOWNE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType8kUNKNOWNE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba16bb40b9102367393ecf8213078d1c6e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUNKNOWN</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType8kUNKNOWNE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelTypeE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelTypeE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ModelType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46a577f62bbac6fac26ae5fcbfbd638405b"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDECODER_ONLY</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46ab9f0d192962e27b51920b525cfc0c8a6"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kENCODER_ONLY</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46a9faae00b617c7bb72d509e32454d1758"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kENCODER_DECODER</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12BatchingTypeE">
<span id="_CPPv3N12tensorrt_llm8executor12BatchingTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor12BatchingTypeE"></span><span class="target" id="types_8h_1aca90eaee2e69172d309f77a581b824e3"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BatchingType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The batching type. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12BatchingType7kSTATICE">
<span id="_CPPv3N12tensorrt_llm8executor12BatchingType7kSTATICE"></span><span id="_CPPv2N12tensorrt_llm8executor12BatchingType7kSTATICE"></span><span class="target" id="types_8h_1aca90eaee2e69172d309f77a581b824e3a201894ba187a596f534f6406676f96bf"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kSTATIC</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12BatchingType7kSTATICE" title="Link to this definition"></a><br /></dt>
<dd><p>STATIC refers to the traditional batching scheme with a batch of requests running in lockstep until the full generation for all of them is complete. Requests in a batch are all padded up to the maximum input and output sequence length of any member of the batch. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE">
<span id="_CPPv3N12tensorrt_llm8executor12BatchingType9kINFLIGHTE"></span><span id="_CPPv2N12tensorrt_llm8executor12BatchingType9kINFLIGHTE"></span><span class="target" id="types_8h_1aca90eaee2e69172d309f77a581b824e3af9f499a6c3b205667d7f5ddba6bf4c02"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINFLIGHT</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE" title="Link to this definition"></a><br /></dt>
<dd><p>INFLIGHT refers to a scheme where newly arrived requests are dynamically incorporated into the batch under execution, and requests are returned as soon as the end condition is met without any padding. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicyE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p>The policy used to select the subset of available requests in each iteration of the executor generation loop. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428ae7df1f3a699ee2bfbe4328f94e825b7e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMAX_UTILIZATION</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE" title="Link to this definition"></a><br /></dt>
<dd><p>MAX_UTILIZATION packs as many requests as the underlying TRT engine can support in any iteration of the InflightBatching generation loop. While this is expected to maximize GPU throughput, it might require that some requests be paused and restarted depending on peak KV cache memory availability. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428a880f95ca896ec21f5e0a981aac04d10b"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGUARANTEED_NO_EVICT</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE" title="Link to this definition"></a><br /></dt>
<dd><p>GUARANTEED_NO_EVICT uses KV cache more conservatively guaranteeing that a request, once started, will run to completion without eviction. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy13kSTATIC_BATCHE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicy13kSTATIC_BATCHE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicy13kSTATIC_BATCHE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428ac52787543a2afbe58114adaf73d0d1b6"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kSTATIC_BATCH</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy13kSTATIC_BATCHE" title="Link to this definition"></a><br /></dt>
<dd><p>kSTATIC_BATCH does not schedule new requests until all requests in current batch are completed. Similar to kGUARANTEED_NO_EVICT, requests will run to completion without eviction. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor21ContextChunkingPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor21ContextChunkingPolicyE"></span><span class="target" id="types_8h_1a6d4f91b57e2ce01f804855083e22a4bf"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE">
<span id="_CPPv3N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE"></span><span id="_CPPv2N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE"></span><span class="target" id="types_8h_1a6d4f91b57e2ce01f804855083e22a4bfac5d100effe3b0eee3f5b4d48d4b5a5a9"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFIRST_COME_FIRST_SERVED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE" title="Link to this definition"></a><br /></dt>
<dd><p>Sequential chunking, complete the unfinished context phase first. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE"></span><span class="target" id="types_8h_1a6d4f91b57e2ce01f804855083e22a4bfac263978419a580b4ef16a199b050f89c"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kEQUAL_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Iterate through each context request in sequence and attempt to increase its chunk count until the constraint is exceeded. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationTypeE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationTypeE"></span><span class="target" id="types_8h_1a93b2af939dbc2cc1690fb10232675ba0"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CommunicationType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationType4kMPIE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationType4kMPIE"></span><span class="target" id="types_8h_1a93b2af939dbc2cc1690fb10232675ba0a4352f81c41aaa6b77e87c062d4347e2b"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMPI</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationModeE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationModeE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationModeE"></span><span class="target" id="types_8h_1ace2966b11ce94befb6438957e8aef1a4"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CommunicationMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationMode7kLEADERE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationMode7kLEADERE"></span><span class="target" id="types_8h_1ace2966b11ce94befb6438957e8aef1a4a211ca40446bff2e65500e3e544664df7"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kLEADER</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE"></span><span class="target" id="types_8h_1ace2966b11ce94befb6438957e8aef1a4a88924e2c20e89e79d5782505f7228158"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kORCHESTRATOR</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStageE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStageE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStageE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097ca"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestStage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="Link to this definition"></a><br /></dt>
<dd><p>Enum class that represents the state of a request. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage7kQUEUEDE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage7kQUEUEDE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage7kQUEUEDE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caaa2247736a3991a2eb2fe91dd90b2f6af"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kQUEUED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage7kQUEUEDE" title="Link to this definition"></a><br /></dt>
<dd><p><a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> that have been received but not yet included in the active requests (due to constraints such as maximum batch size for example). </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caaf777328746e427f4925d2423d7722757"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kENCODER_IN_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request in encoder phase. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caa0d34fe21e6c4aeae3dc8b16b2626ad0a"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCONTEXT_IN_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request in context phase. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caa00590ca88f1c0cfb06a0498c299476d4"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGENERATION_IN_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request in generation phase. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caab437475b0e4a8e154f8af3ba70c51dc3"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGENERATION_COMPLETE</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request for which generation has completed. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReasonE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReasonE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReasonE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FinishReason</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReasonE" title="Link to this definition"></a><br /></dt>
<dd><p>The reason why the model stopped generating tokens for a request. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReason13kNOT_FINISHEDE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReason13kNOT_FINISHEDE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReason13kNOT_FINISHEDE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6a2e35cacb9bb2d1aeeeccade227905c0e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kNOT_FINISHED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReason13kNOT_FINISHEDE" title="Link to this definition"></a><br /></dt>
<dd><p>The request is not finished. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReason7kEND_IDE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReason7kEND_IDE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReason7kEND_IDE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6a8dd28f2cf09133dd1f5faee718454414"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kEND_ID</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReason7kEND_IDE" title="Link to this definition"></a><br /></dt>
<dd><p>The request finished because the end id was generated. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReason11kSTOP_WORDSE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReason11kSTOP_WORDSE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReason11kSTOP_WORDSE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6a2ab216f49c325a41266959adae89d820"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kSTOP_WORDS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReason11kSTOP_WORDSE" title="Link to this definition"></a><br /></dt>
<dd><p>The request finished because a stop word was generated. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReason7kLENGTHE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReason7kLENGTHE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReason7kLENGTHE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6ad9ab945eba83fecb2c324057275c0876"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kLENGTH</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReason7kLENGTHE" title="Link to this definition"></a><br /></dt>
<dd><p>The request finished because the maximum number of tokens was reached. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReason10kTIMED_OUTE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReason10kTIMED_OUTE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReason10kTIMED_OUTE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6ad470dffc98a09c6158718c8827de10b4"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTIMED_OUT</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReason10kTIMED_OUTE" title="Link to this definition"></a><br /></dt>
<dd><p>The request finished because it got timed out (via the mAllotedTime parameter) </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12FinishReason10kCANCELLEDE">
<span id="_CPPv3N12tensorrt_llm8executor12FinishReason10kCANCELLEDE"></span><span id="_CPPv2N12tensorrt_llm8executor12FinishReason10kCANCELLEDE"></span><span class="target" id="types_8h_1a224950d1b5760e0c752649295b84ada6a0db9706b012c1c09997a42ef9fb76791"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCANCELLED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12FinishReason10kCANCELLEDE" title="Link to this definition"></a><br /></dt>
<dd><p>The request was cancelled by calling cancelRequest. </p>
</dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy">
<span id="_CPPv3N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy"></span><span id="_CPPv2N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy"></span><span id="tensorrt_llm::executor::lshift-operator__osR.CapacitySchedulerPolicy"></span><span class="target" id="types_8h_1a7e30017998937539ebe04d46a822ef5b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">policy</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy">
<span id="_CPPv3N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy"></span><span id="_CPPv2N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy"></span><span id="tensorrt_llm::executor::lshift-operator__osR.ContextChunkingPolicy"></span><span class="target" id="types_8h_1a314256f0336f928899256adc3cb97145"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">policy</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIterationE">
<span id="_CPPv3N12tensorrt_llm8executor24DebugTensorsPerIterationE"></span><span id="_CPPv2N12tensorrt_llm8executor24DebugTensorsPerIterationE"></span><span id="tensorrt_llm::executor::DebugTensorsPerIteration"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1DebugTensorsPerIteration"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DebugTensorsPerIteration</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIterationE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the debug tensors in an iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIteration4iterE">
<span id="_CPPv3N12tensorrt_llm8executor24DebugTensorsPerIteration4iterE"></span><span id="_CPPv2N12tensorrt_llm8executor24DebugTensorsPerIteration4iterE"></span><span id="tensorrt_llm::executor::DebugTensorsPerIteration::iter__IterationType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1DebugTensorsPerIteration_1a27303becd7f64dac5c3b36c0830151ee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIteration4iterE" title="Link to this definition"></a><br /></dt>
<dd><p>The iteration id for these tensors. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIteration12debugTensorsE">
<span id="_CPPv3N12tensorrt_llm8executor24DebugTensorsPerIteration12debugTensorsE"></span><span id="_CPPv2N12tensorrt_llm8executor24DebugTensorsPerIteration12debugTensorsE"></span><span id="tensorrt_llm::executor::DebugTensorsPerIteration::debugTensors__std::map:ss.Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1DebugTensorsPerIteration_1acba4a2160ca6f95df41e635d95780a7f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">map</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">debugTensors</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24DebugTensorsPerIteration12debugTensorsE" title="Link to this definition"></a><br /></dt>
<dd><p>The debug tensors for this iteration. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingModeE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingModeE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingModeE"></span><span id="tensorrt_llm::executor::DecodingMode"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>mode of the decoder </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a3448ce5ea22fc30852625ab0fef44d7c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UnderlyingType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">uint32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode14useTemperatureEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode14useTemperatureEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode14useTemperatureEb"></span><span id="tensorrt_llm::executor::DecodingMode::useTemperature__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a53684be976895eebc1664f0d6c70ae6a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useTemperature</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useTemp</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14useTemperatureEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb"></span><span id="tensorrt_llm::executor::DecodingMode::useOccurrencePenalties__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1adbefa253ae6fa8818681b7cb8a4ec0f9"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useOccurrencePenalties</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb"></span><span id="tensorrt_llm::executor::DecodingMode::usePresencePenalty__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ac7788f151b7ad6f02d3accfcbc998d8b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePresencePenalty</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb"></span><span id="tensorrt_llm::executor::DecodingMode::useRepetitionPenalty__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af96b575f7955dc093b069fd43b55a634"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb"></span><span id="tensorrt_llm::executor::DecodingMode::useFrequencyPenalty__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af27a60691183d1fababc834943b0a0da"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12useMinLengthEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12useMinLengthEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12useMinLengthEb"></span><span id="tensorrt_llm::executor::DecodingMode::useMinLength__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a87b550d95c6317042ddafd47919e3dd6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useMinLength</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useMinLen</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useMinLengthEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12useBanTokensEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12useBanTokensEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12useBanTokensEb"></span><span id="tensorrt_llm::executor::DecodingMode::useBanTokens__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1ab86e40f9cac2d7f9bc7f0e289a50c6"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useBanTokens</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">banTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useBanTokensEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode11useBanWordsEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode11useBanWordsEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode11useBanWordsEb"></span><span id="tensorrt_llm::executor::DecodingMode::useBanWords__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a3df6e7d8e5897accecaa4020754f8907"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useBanWords</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">banWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11useBanWordsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb"></span><span id="tensorrt_llm::executor::DecodingMode::useNoRepeatNgramSize__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1afd8cb08a97e7ef3d7d22a2ef9a5da52d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12useStopWordsEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12useStopWordsEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12useStopWordsEb"></span><span id="tensorrt_llm::executor::DecodingMode::useStopWords__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1abb9e70c7d442d0ff4cb7fd4520f5d2ed"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useStopWords</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stopWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useStopWordsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb"></span><span id="tensorrt_llm::executor::DecodingMode::useMaxLengthStop__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a5b185534052f4c92bfbc0656347dc815"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useMaxLengthStop</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxLengthStop</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb"></span><span id="tensorrt_llm::executor::DecodingMode::useExplicitEosStop__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a7619fa8157c8ae7a7f14080d9658b07f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useExplicitEosStop</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">explicitEosStop</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode6isAutoEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode6isAutoEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode6isAutoEv"></span><span id="tensorrt_llm::executor::DecodingMode::isAutoCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a13a8c66dca42a90bbf87f9b1dd2ff975"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isAuto</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isAutoEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopKEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode6isTopKEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode6isTopKEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopKCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1acef6cd75fa3225f59b93bf5b37f5091e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopK</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopKEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode6isTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode6isTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopPCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a6d89a1dc788260a68bb31cf6d914afe7"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopKorTopPCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aa63cc1be02bcc70d2984e362a3eed173"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopKorTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopKandTopPCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a311be6392cb57c618401c85435471a9f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopKandTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv"></span><span id="tensorrt_llm::executor::DecodingMode::isBeamSearchCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aca93dead027944e2dce0aee783af1805"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isBeamSearch</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode8isMedusaEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode8isMedusaEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode8isMedusaEv"></span><span id="tensorrt_llm::executor::DecodingMode::isMedusaCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a918a24ee9837d1d4445f993f58e87473"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isMedusa</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8isMedusaEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv"></span><span id="tensorrt_llm::executor::DecodingMode::isLookaheadCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1c23e7c880b6b756654aada19882be64"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLookahead</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::isExplicitDraftTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a5cf22aa39532d314b197f32243506ca7"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isExplicitDraftTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExternalDraftTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode21isExternalDraftTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode21isExternalDraftTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::isExternalDraftTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a9369abde5d01be56b863019dd4566372"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isExternalDraftTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExternalDraftTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode7isEagleEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode7isEagleEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode7isEagleEv"></span><span id="tensorrt_llm::executor::DecodingMode::isEagleCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aacea10c88d6ed17b9e0afb30ba25c93c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isEagle</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode7isEagleEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseTemperatureCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1f9428f57c22e7f0ef830093c871443c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseTemperature</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUsePresencePenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a117bd359894342c3d47f5aced212c8e0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUsePresencePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseFrequencyPenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1cffde64810926fefcbd62248dc57f13"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseRepetitionPenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a0f1b4906c1e8da7da08655e9d95460bb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseMinLengthCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ab570344af6e79189256fdd2990de50f1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseMinLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseOccurrencePenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a3366394589b741680cf971361e3b1a03"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseOccurrencePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUsePenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a46f7c33f03d6bc54569f0b5ff74d64fd"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUsePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseBanWordsCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1abdca9b5b5b708624d9a07a5530492c3c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseBanWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseNoRepeatNgramSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ab521096f583e2d773acad9d86e1d1925"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseBanTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1acc4cf4d789890aa53d21de141d9e898d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseBanTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseStopWordsCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a943fa69829d69a7158c836daaa232e77"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseStopWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseMaxLengthStopCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ada5840f7445fc38d13912b6c7b9806bd"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseMaxLengthStop</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseExplicitEosStopCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aad01d4a47d716f65ed97dcace6d00d75"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseExplicitEosStop</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseStopCriteriaCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a75e451f8c7cff66006074a2e5aa9c493"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseStopCriteria</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode"></span><span id="tensorrt_llm::executor::DecodingMode::eq-operator__DecodingModeCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aaa854d9caaad142db1f8f0cbfcba8f8f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType"></span><span id="tensorrt_llm::executor::DecodingMode::DecodingMode__UnderlyingTypeCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a844ac72db3cf0d3961dc9a90f86e101b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingMode</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode8getStateEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode8getStateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode8getStateEv"></span><span id="tensorrt_llm::executor::DecodingMode::getStateCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a27d6ca8f5f71b63b26dc7eb2b3d03cfc"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getState</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8getStateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode4AutoEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode4AutoEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode4AutoEv"></span><span id="tensorrt_llm::executor::DecodingMode::AutoCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ae19b52345b0ceaa53893604fffd8f0e9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Auto</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4AutoEv" title="Link to this definition"></a><br /></dt>
<dd><p>No mode specified. Config will be determined from the beam width of the first request at runtime TopKTopP if beamWidth == 1, BeamSearch otherwise. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode4TopKEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode4TopKEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode4TopKEv"></span><span id="tensorrt_llm::executor::DecodingMode::TopKCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aa373bed34ebe0279642f5c040e531d2d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TopK</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopKEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode4TopPEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode4TopPEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode4TopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::TopPCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a06af714392ace9ed4a8e12a929f90436"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode8TopKTopPEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode8TopKTopPEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode8TopKTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::TopKTopPCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a6357866b6b060de43e847377ced8cfab"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TopKTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8TopKTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode10BeamSearchEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode10BeamSearchEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode10BeamSearchEv"></span><span id="tensorrt_llm::executor::DecodingMode::BeamSearchCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a37f194965847988e8a432ada9d22de66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BeamSearch</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10BeamSearchEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode6MedusaEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode6MedusaEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode6MedusaEv"></span><span id="tensorrt_llm::executor::DecodingMode::MedusaCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a534c86f3ba680562b04f0a859027e019"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Medusa</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6MedusaEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode9LookaheadEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode9LookaheadEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode9LookaheadEv"></span><span id="tensorrt_llm::executor::DecodingMode::LookaheadCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1abfa9bfe2f07541d845c8c4e9967eeac7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Lookahead</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9LookaheadEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::ExplicitDraftTokensCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a19ec77f227684778f24dceb21413a8ac"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExplicitDraftTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19ExternalDraftTokensEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19ExternalDraftTokensEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19ExternalDraftTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::ExternalDraftTokensCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aa640976fea3f031ebac9d768f522df91"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExternalDraftTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19ExternalDraftTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5EagleEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5EagleEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5EagleEv"></span><span id="tensorrt_llm::executor::DecodingMode::EagleCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a3a3b86b6930f3a6f29070f25bcfa0510"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Eagle</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5EagleEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType"></span><span id="tensorrt_llm::executor::DecodingMode::anyBitSet__UnderlyingTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a32ede164c41b093aae7c85b3219e5f17"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">anyBitSet</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType"></span><span id="tensorrt_llm::executor::DecodingMode::allBitSet__UnderlyingTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a26b5b3ca9c8a51fe1c80634d88467fdf"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allBitSet</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb"></span><span id="tensorrt_llm::executor::DecodingMode::setBitTo__UnderlyingType.bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a6d9fd972c8e6732431571cf413d9eb96"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBitTo</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">state</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">x</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode6mStateE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode6mStateE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode6mStateE"></span><span id="tensorrt_llm::executor::DecodingMode::mState__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a7d858720e8c48b761bfb6fae613cd69f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mState</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6mStateE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-attributes">Private Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseRepetitionPenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a56fd74b99c930baabf3cec5ee5064781"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseRepetitionPenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseFrequencyPenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a87b52408c776e9e46b80cfc53973d131"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseFrequencyPenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUsePresencePenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a62bbed2ed4e23bd133a50b8478f0a340"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUsePresencePenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseTemperature__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a68b45f890a7ce1a6c690b6450bd90775"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseTemperature</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">3</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseMinLength__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a505e61c877fb6e1a0f55bd97e190c387"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseMinLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">4</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseBanWords__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a99e799e22a27fae5470f85b56d3b4350"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseBanWords</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">5</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseStopWords__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ae0b84d84d79681471b066d0ef57ee186"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseStopWords</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">6</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseMaxLengthStop__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aca8d72e75f44aa7884f8201bef2c9b13"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseMaxLengthStop</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">7</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseExplicitEosStop__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a50b13f8199ae42fbd7889f04acb4bd50"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseExplicitEosStop</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseNoRepeatNgramSize__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ae715ea3674c97b0d768c779cb763866d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseNoRepeatNgramSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">9</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE"></span><span id="tensorrt_llm::executor::DecodingMode::kStandardStopCriteria__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af3f05360268a782c23865365481c5b09"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kStandardStopCriteria</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE" title="tensorrt_llm::executor::DecodingMode::kUseStopWords"><span class="n"><span class="pre">kUseStopWords</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE" title="tensorrt_llm::executor::DecodingMode::kUseMaxLengthStop"><span class="n"><span class="pre">kUseMaxLengthStop</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseOccurrencePenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ab96b961e00ac3efd8cea889aac42e7ed"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseOccurrencePenalties</span></span></span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUseRepetitionPenalties"><span class="n"><span class="pre">kUseRepetitionPenalties</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUseFrequencyPenalties"><span class="n"><span class="pre">kUseFrequencyPenalties</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUsePresencePenalties"><span class="n"><span class="pre">kUsePresencePenalties</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUsePenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1f64b7fd17705346396cc54b8e0898ea"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUsePenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUseOccurrencePenalties"><span class="n"><span class="pre">kUseOccurrencePenalties</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE" title="tensorrt_llm::executor::DecodingMode::kUseTemperature"><span class="n"><span class="pre">kUseTemperature</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE" title="tensorrt_llm::executor::DecodingMode::kUseMinLength"><span class="n"><span class="pre">kUseMinLength</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseBanTokens__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aec58c8d8748972a10f900b0cd2899afa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseBanTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE" title="tensorrt_llm::executor::DecodingMode::kUseNoRepeatNgramSize"><span class="n"><span class="pre">kUseNoRepeatNgramSize</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE" title="tensorrt_llm::executor::DecodingMode::kUseBanWords"><span class="n"><span class="pre">kUseBanWords</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode9kNumFlagsE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode9kNumFlagsE"></span><span id="tensorrt_llm::executor::DecodingMode::kNumFlags__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a4ccbf3895859431cc61a3390a2ea77fb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kNumFlags</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">10</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5kAutoE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5kAutoE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5kAutoE"></span><span id="tensorrt_llm::executor::DecodingMode::kAuto__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ab447608fe495937145568cf45fedaf8e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kAuto</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kAutoE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5kTopKE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5kTopKE"></span><span id="tensorrt_llm::executor::DecodingMode::kTopK__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1b4479a1017694d10148f310055f3d19"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTopK</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5kTopPE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5kTopPE"></span><span id="tensorrt_llm::executor::DecodingMode::kTopP__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a321db8b7aa9a93a6bb2af6580d060b6e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTopP</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode11kBeamSearchE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode11kBeamSearchE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode11kBeamSearchE"></span><span id="tensorrt_llm::executor::DecodingMode::kBeamSearch__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a78a537977a0d6c09d510c3442d389647"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBeamSearch</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">3</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11kBeamSearchE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode7kMedusaE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode7kMedusaE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode7kMedusaE"></span><span id="tensorrt_llm::executor::DecodingMode::kMedusa__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a7e480511b5f4e67bfae949ad42d3a7d1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMedusa</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">4</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode7kMedusaE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode10kLookaheadE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode10kLookaheadE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode10kLookaheadE"></span><span id="tensorrt_llm::executor::DecodingMode::kLookahead__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aaae72084fe5bb0a0cf7a93acee8ac0ba"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kLookahead</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">5</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10kLookaheadE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE"></span><span id="tensorrt_llm::executor::DecodingMode::kExplicitDraftTokens__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a23bc169569946f8eb079164d0d39ab0b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kExplicitDraftTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">6</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20kExternalDraftTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20kExternalDraftTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20kExternalDraftTokensE"></span><span id="tensorrt_llm::executor::DecodingMode::kExternalDraftTokens__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a280281eda5b35b040985638b2228ac35"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kExternalDraftTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">7</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20kExternalDraftTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode6kEagleE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode6kEagleE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode6kEagleE"></span><span id="tensorrt_llm::executor::DecodingMode::kEagle__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a9c932c52575cc4cc3949de88f9a66d07"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kEagle</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6kEagleE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode9kTopKTopPE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode9kTopKTopPE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode9kTopKTopPE"></span><span id="tensorrt_llm::executor::DecodingMode::kTopKTopP__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a8f4f53c2825bdbee88983507f0c2f413"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTopKTopP</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE" title="tensorrt_llm::executor::DecodingMode::kTopK"><span class="n"><span class="pre">kTopK</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE" title="tensorrt_llm::executor::DecodingMode::kTopP"><span class="n"><span class="pre">kTopP</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kTopKTopPE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE">
<span id="_CPPv3N12tensorrt_llm8executor22DisServingRequestStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor22DisServingRequestStatsE"></span><span id="tensorrt_llm::executor::DisServingRequestStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1DisServingRequestStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DisServingRequestStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the request stats in the case of disaggregated serving. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22DisServingRequestStats17kvCacheTransferMSE">
<span id="_CPPv3N12tensorrt_llm8executor22DisServingRequestStats17kvCacheTransferMSE"></span><span id="_CPPv2N12tensorrt_llm8executor22DisServingRequestStats17kvCacheTransferMSE"></span><span id="tensorrt_llm::executor::DisServingRequestStats::kvCacheTransferMS__double"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1DisServingRequestStats_1a5f3f1be5b29fb05aea1f8bd97b9ce0e6"></span><span class="kt"><span class="pre">double</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheTransferMS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStats17kvCacheTransferMSE" title="Link to this definition"></a><br /></dt>
<dd><p>The total time spent on transferring KV cache from context phase to generation phase (ms) </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStatsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">InflightBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of inflight batching models for a single iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numScheduledRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1adf6843f2538709caad5542216b13693e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numScheduledRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of scheduled requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numContextRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1afc34637bdddb02cb101f38a9c8a4a50a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numContextRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests in context stage. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numGenRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a226f8ec68000216bfeedc6040e08da7d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGenRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests in generation stage. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numPausedRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a942c50f8eaba7dd00ded9f10e257f286"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numPausedRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of paused requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numCtxTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a052d62534092ccc9824332a1d55da3a8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCtxTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of context tokens in the iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::microBatchId__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a945be330caa27de5a173aec6d71a6b03"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">microBatchId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE" title="Link to this definition"></a><br /></dt>
<dd><p>Index of mirco batch. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::avgNumDecodedTokensPerIter__float"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1acca77d655808fe504c6f99c8c6e9c456"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">avgNumDecodedTokensPerIter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE" title="Link to this definition"></a><br /></dt>
<dd><p>Average number of tokens decoded per request per iteration. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStatsE"></span><span id="tensorrt_llm::executor::IterationStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IterationStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a single iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats9timestampE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats9timestampE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats9timestampE"></span><span id="tensorrt_llm::executor::IterationStats::timestamp__ss"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1af1d7e6fb431b3bae3be69d0cc3a6b00e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">timestamp</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats9timestampE" title="Link to this definition"></a><br /></dt>
<dd><p>Ending time of this iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats4iterE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats4iterE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats4iterE"></span><span id="tensorrt_llm::executor::IterationStats::iter__IterationType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ae19cff294da29d71592cabdc54be9774"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats4iterE" title="Link to this definition"></a><br /></dt>
<dd><p>Iteration id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats13iterLatencyMSE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats13iterLatencyMSE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats13iterLatencyMSE"></span><span id="tensorrt_llm::executor::IterationStats::iterLatencyMS__double"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a08d812da19f047a777e11e217b82eaa8"></span><span class="kt"><span class="pre">double</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iterLatencyMS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats13iterLatencyMSE" title="Link to this definition"></a><br /></dt>
<dd><p>Iteration latency (ms) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats31newActiveRequestsQueueLatencyMSE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats31newActiveRequestsQueueLatencyMSE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats31newActiveRequestsQueueLatencyMSE"></span><span id="tensorrt_llm::executor::IterationStats::newActiveRequestsQueueLatencyMS__double"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac5d8644864440b8d0208ac6e0946025e"></span><span class="kt"><span class="pre">double</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">newActiveRequestsQueueLatencyMS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats31newActiveRequestsQueueLatencyMSE" title="Link to this definition"></a><br /></dt>
<dd><p>The total time spent in queue by the requests that became active in this iteration (ms) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats20numNewActiveRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats20numNewActiveRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats20numNewActiveRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::numNewActiveRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac27bf9ead5c27b5b15d390b57919b966"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numNewActiveRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20numNewActiveRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of new fetched active requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats17numActiveRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats17numActiveRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats17numActiveRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::numActiveRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a7b8b4c8acde99a1eb3de70050e770458"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numActiveRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numActiveRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of active requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::numQueuedRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1abddd1933dca91cbf6336b10e3fdf6e1b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numQueuedRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of queued requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats20numCompletedRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats20numCompletedRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats20numCompletedRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::numCompletedRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ae7a03c208c003d49e477c90b5101c4cc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCompletedRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20numCompletedRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests that were completed in this iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::maxNumActiveRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac8c5c8bf27f8c0eec9fa7ae41da3fe67"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumActiveRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of max active requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats18maxBatchSizeStaticE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats18maxBatchSizeStaticE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats18maxBatchSizeStaticE"></span><span id="tensorrt_llm::executor::IterationStats::maxBatchSizeStatic__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a2916efcce10da5d182bb478c6ab26182"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxBatchSizeStatic</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats18maxBatchSizeStaticE" title="Link to this definition"></a><br /></dt>
<dd><p>Static max batch size passed to the executor. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats28maxBatchSizeTunerRecommendedE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats28maxBatchSizeTunerRecommendedE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats28maxBatchSizeTunerRecommendedE"></span><span id="tensorrt_llm::executor::IterationStats::maxBatchSizeTunerRecommended__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a13cd7271b5c9263e800f452eb998a1f7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxBatchSizeTunerRecommended</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats28maxBatchSizeTunerRecommendedE" title="Link to this definition"></a><br /></dt>
<dd><p>Batch size produced by dynamic tuner based on input stats. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats19maxBatchSizeRuntimeE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats19maxBatchSizeRuntimeE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats19maxBatchSizeRuntimeE"></span><span id="tensorrt_llm::executor::IterationStats::maxBatchSizeRuntime__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a2e21e17176e117804ea1702ba1a44041"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxBatchSizeRuntime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19maxBatchSizeRuntimeE" title="Link to this definition"></a><br /></dt>
<dd><p>&#64;brife The min of maxBatchSizeStatic and maxBatchSizeRuntimeUpperbound </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats18maxNumTokensStaticE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats18maxNumTokensStaticE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats18maxNumTokensStaticE"></span><span id="tensorrt_llm::executor::IterationStats::maxNumTokensStatic__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1abade518ffdca3b4d0f8199d3174d9785"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumTokensStatic</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats18maxNumTokensStaticE" title="Link to this definition"></a><br /></dt>
<dd><p>&#64;brife Static max num tokens passed to the executor </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats28maxNumTokensTunerRecommendedE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats28maxNumTokensTunerRecommendedE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats28maxNumTokensTunerRecommendedE"></span><span id="tensorrt_llm::executor::IterationStats::maxNumTokensTunerRecommended__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a1e0da1bdf37481742d2edea8f12e223f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumTokensTunerRecommended</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats28maxNumTokensTunerRecommendedE" title="Link to this definition"></a><br /></dt>
<dd><p>&#64;brife Max num tokens produced by dynamic tuner based on input stats </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats19maxNumTokensRuntimeE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats19maxNumTokensRuntimeE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats19maxNumTokensRuntimeE"></span><span id="tensorrt_llm::executor::IterationStats::maxNumTokensRuntime__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ae24f995f04e83c7135a9add67b135c9b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumTokensRuntime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19maxNumTokensRuntimeE" title="Link to this definition"></a><br /></dt>
<dd><p>&#64;brife The runtime max num tokens </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats11gpuMemUsageE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats11gpuMemUsageE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats11gpuMemUsageE"></span><span id="tensorrt_llm::executor::IterationStats::gpuMemUsage__s"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac975025c2e40a167051e4c79afcea908"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpuMemUsage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11gpuMemUsageE" title="Link to this definition"></a><br /></dt>
<dd><p>GPU memory usage in bytes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats11cpuMemUsageE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats11cpuMemUsageE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats11cpuMemUsageE"></span><span id="tensorrt_llm::executor::IterationStats::cpuMemUsage__s"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1abba95b563f5fc8459b626b05472eef38"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpuMemUsage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11cpuMemUsageE" title="Link to this definition"></a><br /></dt>
<dd><p>CPU memory usage in bytes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE"></span><span id="tensorrt_llm::executor::IterationStats::pinnedMemUsage__s"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a99f299d90d3366ae1f694890fd859455"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinnedMemUsage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE" title="Link to this definition"></a><br /></dt>
<dd><p>Pinned memory usage in bytes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats12kvCacheStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats12kvCacheStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats12kvCacheStatsE"></span><span id="tensorrt_llm::executor::IterationStats::kvCacheStats__std::optional:KvCacheStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac234045a5fdd46880205b99f6d326977"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats12kvCacheStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to KV caches. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE"></span><span id="tensorrt_llm::executor::IterationStats::crossKvCacheStats__std::optional:KvCacheStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a3f985a83e7ba30a463c13c4cafff87fe"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">crossKvCacheStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to cross KV caches. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE"></span><span id="tensorrt_llm::executor::IterationStats::staticBatchingStats__std::optional:StaticBatchingStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a361f059648067ed27ccd1baa91656e8f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">staticBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to static batching. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE"></span><span id="tensorrt_llm::executor::IterationStats::inflightBatchingStats__std::optional:InflightBatchingStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a8a9f5f9365d9c7cb7e2adcb5e76fb811"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">inflightBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to inflight batching. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStatsE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStatsE"></span><span id="tensorrt_llm::executor::KvCacheStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a KV cache manager. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::maxNumBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a2952c746c5724a096a0f90037cc95a9b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Max number of blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::freeNumBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a4393a6d986ba543b215c761888732ddc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">freeNumBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of free blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::usedNumBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a9cd563d7a3d9a49a7bea72fbb386fd0c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usedNumBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of used blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE"></span><span id="tensorrt_llm::executor::KvCacheStats::tokensPerBlock__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a7b0c51dae64cd0af386fb0f0104dedc8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tokensPerBlock</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of tokens per block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::allocTotalBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a88f44cd389cb3813a93245cab0aa96d8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocTotalBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of total allocated block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::allocNewBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a804a58fa6ad301b7ddfe922cb413d265"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocNewBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of newly allocated block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::reusedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a1c54f95634272634dcb7779bfd87e162"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reusedBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of reused block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats12missedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats12missedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats12missedBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::missedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1aeb9ea6e480694674dceed68166950f13"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">missedBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12missedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of not reused block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats12cacheHitRateE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats12cacheHitRateE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats12cacheHitRateE"></span><span id="tensorrt_llm::executor::KvCacheStats::cacheHitRate__float"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a85799d137b58b9b8b9020f9b4e9763f6"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cacheHitRate</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12cacheHitRateE" title="Link to this definition"></a><br /></dt>
<dd><p>Measuring the KV Cache reuse rate. cacheHitRate = reusedBlocks / (reusedBlocks + missedBlocks). </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetricsE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestPerfMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetricsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a request. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1af3b6e1baa7b1613f81570f8541ea18ee"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TimePoint</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">time_point</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">steady_clock</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13timingMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13timingMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13timingMetricsE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::timingMetrics__TimingMetrics"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1a7ca683e5e7da1cf81d52583af9fd24e8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics"><span class="n"><span class="pre">TimingMetrics</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">timingMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13timingMetricsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14kvCacheMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14kvCacheMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14kvCacheMetricsE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::kvCacheMetrics__KvCacheMetrics"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1a79c3d18a5665397fb4750ee7c434cec2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetricsE" title="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics"><span class="n"><span class="pre">KvCacheMetrics</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14kvCacheMetricsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9firstIterE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics9firstIterE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics9firstIterE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::firstIter__std::optional:IterationType:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1a39d77233c4e9972b4483bf2e479e8a07"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">firstIter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9firstIterE" title="Link to this definition"></a><br /></dt>
<dd><p>First iteration where the request was processed. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics8lastIterE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics8lastIterE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics8lastIterE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::lastIter__std::optional:IterationType:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1adbb66032d68985474ae89d2289a39785"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lastIter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics8lastIterE" title="Link to this definition"></a><br /></dt>
<dd><p>Last iteration where a token was generated. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics4iterE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics4iterE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics4iterE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::iter__std::optional:IterationType:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1ae690ff374e4407ab54ccea9f3a5c560a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics4iterE" title="Link to this definition"></a><br /></dt>
<dd><p>Current iteration. </p>
</dd></dl>
</div>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetricsE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1KvCacheMetrics"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetricsE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics23numTotalAllocatedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics23numTotalAllocatedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics23numTotalAllocatedBlocksE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics::numTotalAllocatedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1KvCacheMetrics_1abe5d5e7ff95f4ea2df6b41acebf6a1ec"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numTotalAllocatedBlocks</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics23numTotalAllocatedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of total allocated blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics21numNewAllocatedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics21numNewAllocatedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics21numNewAllocatedBlocksE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics::numNewAllocatedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1KvCacheMetrics_1aba5ba413272bc8ee7b1692cc829c46ed"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numNewAllocatedBlocks</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics21numNewAllocatedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of newly allocated blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numReusedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numReusedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numReusedBlocksE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics::numReusedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1KvCacheMetrics_1ac73cc1e473ddc919c5b87c5ed2fccc2c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numReusedBlocks</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numReusedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of reused blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numMissedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numMissedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numMissedBlocksE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics::numMissedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1KvCacheMetrics_1abebf60af8d299a1ab57e621cda279070"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numMissedBlocks</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics15numMissedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of missed blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics14kvCacheHitRateE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics14kvCacheHitRateE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics14kvCacheHitRateE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::KvCacheMetrics::kvCacheHitRate__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1KvCacheMetrics_1a42032ed4cdbc9d2af330185bdeb26c3f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheHitRate</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics14KvCacheMetrics14kvCacheHitRateE" title="Link to this definition"></a><br /></dt>
<dd><p>KV Cache Hit Rate, defined as reusedBlocks / (reusedBlocks + missedBlocks) </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetricsE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetricsE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetricsE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TimingMetrics</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetricsE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics11arrivalTimeE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics11arrivalTimeE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics11arrivalTimeE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics::arrivalTime__TimePoint"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics_1abe219c1af2f20d8d2316d3aac6f04565"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">arrivalTime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics11arrivalTimeE" title="Link to this definition"></a><br /></dt>
<dd><p>The time when the request arrived. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18firstScheduledTimeE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18firstScheduledTimeE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18firstScheduledTimeE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics::firstScheduledTime__TimePoint"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics_1a889fa5895662c4c08a86084e868e8a23"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">firstScheduledTime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18firstScheduledTimeE" title="Link to this definition"></a><br /></dt>
<dd><p>The time when the request was first scheduled. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics14firstTokenTimeE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics14firstTokenTimeE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics14firstTokenTimeE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics::firstTokenTime__TimePoint"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics_1ab1f8b72554c8029fa21adfdd9d4c90f0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">firstTokenTime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics14firstTokenTimeE" title="Link to this definition"></a><br /></dt>
<dd><p>The time when the first token was generated. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics13lastTokenTimeE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics13lastTokenTimeE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics13lastTokenTimeE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics::lastTokenTime__TimePoint"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics_1a1a8512ed1362583afbc951fa61af87df"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">lastTokenTime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics13lastTokenTimeE" title="Link to this definition"></a><br /></dt>
<dd><p>The time when the request was finished. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics20kvCacheTransferStartE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics20kvCacheTransferStartE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics20kvCacheTransferStartE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics::kvCacheTransferStart__TimePoint"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics_1add1d41fa479c923306bf6c32b65b5d08"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheTransferStart</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics20kvCacheTransferStartE" title="Link to this definition"></a><br /></dt>
<dd><p>Start time of the KV cache transfer for disaggregated serving. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18kvCacheTransferEndE">
<span id="_CPPv3N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18kvCacheTransferEndE"></span><span id="_CPPv2N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18kvCacheTransferEndE"></span><span id="tensorrt_llm::executor::RequestPerfMetrics::TimingMetrics::kvCacheTransferEnd__TimePoint"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestPerfMetrics_1_1TimingMetrics_1a6c1e9740382c252ab3f819c55fcaf1ac"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics9TimePointE" title="tensorrt_llm::executor::RequestPerfMetrics::TimePoint"><span class="n"><span class="pre">TimePoint</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheTransferEnd</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18RequestPerfMetrics13TimingMetrics18kvCacheTransferEndE" title="Link to this definition"></a><br /></dt>
<dd><p>End time of the KV cache transfer for disaggregated serving. </p>
</dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStatsE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStatsE"></span><span id="tensorrt_llm::executor::RequestStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a single request. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats2idE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats2idE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats2idE"></span><span id="tensorrt_llm::executor::RequestStats::id__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ae069f9ba9675130e5c3e3b9129d2aff1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">id</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats2idE" title="Link to this definition"></a><br /></dt>
<dd><p>The request id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats5stageE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats5stageE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats5stageE"></span><span id="tensorrt_llm::executor::RequestStats::stage__RequestStage"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a6d35e8a95e6cccaceae38961edd5a354"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="tensorrt_llm::executor::RequestStage"><span class="n"><span class="pre">RequestStage</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats5stageE" title="Link to this definition"></a><br /></dt>
<dd><p>The current stage the request is in. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE"></span><span id="tensorrt_llm::executor::RequestStats::contextPrefillPosition__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ae2372e9fb63311f9f95f77cdf8d09f5b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextPrefillPosition</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE" title="Link to this definition"></a><br /></dt>
<dd><p>If using chunked context, the current context prefill position. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE"></span><span id="tensorrt_llm::executor::RequestStats::numGeneratedTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a18f2badb3c0c81ffee60f7f8e22d2b99"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGeneratedTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The number of generated tokens so far. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE"></span><span id="tensorrt_llm::executor::RequestStats::avgNumDecodedTokensPerIter__float"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a44573f7d4ab8fce4a931c661929deab1"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">avgNumDecodedTokensPerIter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE" title="Link to this definition"></a><br /></dt>
<dd><p>The average number of decoded tokens per iteration. It is &gt;= 1 for speculative decoding. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats9scheduledE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats9scheduledE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats9scheduledE"></span><span id="tensorrt_llm::executor::RequestStats::scheduled__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ab928aa93fcf7a22e40adfd10a62aee04"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">scheduled</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats9scheduledE" title="Link to this definition"></a><br /></dt>
<dd><p>Whether the request is scheduled for the current iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats6pausedE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats6pausedE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats6pausedE"></span><span id="tensorrt_llm::executor::RequestStats::paused__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a6f74bde53fc8910bad82297f8f741deb"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">paused</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats6pausedE" title="Link to this definition"></a><br /></dt>
<dd><p>Whether the request is being paused at the current iteration due to lack of resources (KV cache blocks exhaustion for example) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats15disServingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats15disServingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats15disServingStatsE"></span><span id="tensorrt_llm::executor::RequestStats::disServingStats__std::optional:DisServingRequestStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a85783b94d34f9c189e0fc7033704298b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22DisServingRequestStatsE" title="tensorrt_llm::executor::DisServingRequestStats"><span class="n"><span class="pre">DisServingRequestStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">disServingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats15disServingStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to disaggregated serving. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats26allocTotalBlocksPerRequestE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats26allocTotalBlocksPerRequestE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats26allocTotalBlocksPerRequestE"></span><span id="tensorrt_llm::executor::RequestStats::allocTotalBlocksPerRequest__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ad619ee70e25193236d62493874e97d05"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocTotalBlocksPerRequest</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats26allocTotalBlocksPerRequestE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of total allocated blocks per request. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats24allocNewBlocksPerRequestE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats24allocNewBlocksPerRequestE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats24allocNewBlocksPerRequestE"></span><span id="tensorrt_llm::executor::RequestStats::allocNewBlocksPerRequest__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a1eed06ea5d8edb76e9bc5d6430a46f5e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocNewBlocksPerRequest</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats24allocNewBlocksPerRequestE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of newly allocated blocks per request. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats22reusedBlocksPerRequestE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats22reusedBlocksPerRequestE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats22reusedBlocksPerRequestE"></span><span id="tensorrt_llm::executor::RequestStats::reusedBlocksPerRequest__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a4ad8bcc4dd5f94ad97f353ca06e3849d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reusedBlocksPerRequest</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22reusedBlocksPerRequestE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of reused blocks per request. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats22missedBlocksPerRequestE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats22missedBlocksPerRequestE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats22missedBlocksPerRequestE"></span><span id="tensorrt_llm::executor::RequestStats::missedBlocksPerRequest__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a11599df8fa9ba9f451a05c2ccc1f4a6b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">missedBlocksPerRequest</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22missedBlocksPerRequestE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of missed blocks per request. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats24kvCacheHitRatePerRequestE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats24kvCacheHitRatePerRequestE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats24kvCacheHitRatePerRequestE"></span><span id="tensorrt_llm::executor::RequestStats::kvCacheHitRatePerRequest__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a824d0cd65628a08c3caa05ef3e4187f4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheHitRatePerRequest</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats24kvCacheHitRatePerRequestE" title="Link to this definition"></a><br /></dt>
<dd><p>KV Cache Hit Rate per request, defined as reusedBlocks / (reusedBlocks + missedBlocks) </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE">
<span id="_CPPv3N12tensorrt_llm8executor24RequestStatsPerIterationE"></span><span id="_CPPv2N12tensorrt_llm8executor24RequestStatsPerIterationE"></span><span id="tensorrt_llm::executor::RequestStatsPerIteration"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStatsPerIteration"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of all requests in an iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration4iterE">
<span id="_CPPv3N12tensorrt_llm8executor24RequestStatsPerIteration4iterE"></span><span id="_CPPv2N12tensorrt_llm8executor24RequestStatsPerIteration4iterE"></span><span id="tensorrt_llm::executor::RequestStatsPerIteration::iter__IterationType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStatsPerIteration_1a089fa8d28441ffb3d0f37bd046329ac7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration4iterE" title="Link to this definition"></a><br /></dt>
<dd><p>The iteration id for these stats. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE">
<span id="_CPPv3N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE"></span><span id="tensorrt_llm::executor::RequestStatsPerIteration::requestStats__std::vector:RequestStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStatsPerIteration_1af9fbfa79e3985aa6f824b1d7edf934d0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">requestStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>The stats of all active requests for this iteration. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStatsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StaticBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of static batching models for a single iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numScheduledRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a85a29e377cc0afc1fa3b5ac5e2426509"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numScheduledRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of scheduled requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numContextRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a44b1d7847625f6042de40ef2a5c8ec70"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numContextRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests in context stage. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numCtxTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1aed7aa230c825577f4acb9d43a6f5176d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCtxTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of context tokens in the iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numGenTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a48522c73455172970cb9dc292e53fcd3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGenTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of tokens to generate in the iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::emptyGenSlots__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a3316815cfbe07bb2a486de89b7024c41"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">emptyGenSlots</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of unused generation token slots. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0_bEN12tensorrt_llm8executor10TypeTraitsE">
<span id="_CPPv3I0_bEN12tensorrt_llm8executor10TypeTraitsE"></span><span id="_CPPv2I0_bEN12tensorrt_llm8executor10TypeTraitsE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><a class="headerlink" href="#_CPPv4I0_bEN12tensorrt_llm8executor10TypeTraitsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>For converting a C++ data type to a <code class="docutils literal notranslate"><span class="pre">TrtLmmDataType</span></code>. </p>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIbEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsIbEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsIbEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01bool_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIbEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsIbE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsIbE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsIbE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:b:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01bool_01_4_1a0cf040c47d93165f569715a3e02f22ac"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE" title="tensorrt_llm::executor::DataType::kBOOL"><span class="n"><span class="pre">kBOOL</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIbE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIfEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsIfEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsIfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01float_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIfEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsIfE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsIfE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsIfE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:float:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01float_01_4_1aa3c4709453f3b513d302e3b2e843a464"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP32E" title="tensorrt_llm::executor::DataType::kFP32"><span class="n"><span class="pre">kFP32</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIfE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsI4halfEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsI4halfEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsI4halfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01half_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">half</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsI4halfEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:half:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01half_01_4_1a8b0cc854df1f430f8e8eea8f3b8a733b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP16E" title="tensorrt_llm::executor::DataType::kFP16"><span class="n"><span class="pre">kFP16</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int32_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19PhonyNameDueToError5valueE">
<span id="_CPPv3N12tensorrt_llm8executor19PhonyNameDueToError5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor19PhonyNameDueToError5valueE"></span><span id="tensorrt_llm::executor::PhonyNameDueToError::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4_1a506e37fcee8102b90f320257e12ec485"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E" title="tensorrt_llm::executor::DataType::kINT32"><span class="n"><span class="pre">kINT32</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19PhonyNameDueToError5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int64_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4_1ae1e174a531a0abf5c31ed44525da252d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="tensorrt_llm::executor::DataType::kINT64"><span class="n"><span class="pre">kINT64</span></span></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4_1aaed76141f49476650f91cfbec3367e13"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E" title="tensorrt_llm::executor::DataType::kINT8"><span class="n"><span class="pre">kINT8</span></span></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4_1a36e64efaf2f35f9e9d97c62fb4e74f84"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E" title="tensorrt_llm::executor::DataType::kUINT8"><span class="n"><span class="pre">kUINT8</span></span></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE">
<span id="_CPPv3I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01T_01_5_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE" title="tensorrt_llm::executor::TypeTraits&lt;T*&gt;::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:TP:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01T_01_5_01_4_1a919c622815440975c53c2ba6f5e02c34"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="tensorrt_llm::executor::DataType::kINT64"><span class="n"><span class="pre">kINT64</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="version-h">
<h2>version.h<a class="headerlink" href="#version-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-variables">Variables</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19kTensorRtLlmVersionE">
<span id="_CPPv3N12tensorrt_llm8executor19kTensorRtLlmVersionE"></span><span id="_CPPv2N12tensorrt_llm8executor19kTensorRtLlmVersionE"></span><span id="tensorrt_llm::executor::kTensorRtLlmVersion__auto"></span><span class="target" id="version_8h_1a56d04b7cc852621bcfb54af652ddd6b3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTensorRtLlmVersion</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;0.14.0.dev2024100800&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19kTensorRtLlmVersionE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="../python-api/tensorrt_llm.runtime.html" class="btn btn-neutral float-left" title="Runtime" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="runtime.html" class="btn btn-neutral float-right" title="Runtime" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<jinja2.runtime.BlockReference object at 0x7da74e6b5be0>
<div class="footer">
<p>
Copyright © 2024 NVIDIA Corporation
</p>
<p>
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/" target="_blank" rel="noopener"
data-cms-ai="0">Privacy Policy</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-center/" target="_blank" rel="noopener"
data-cms-ai="0">Manage My Privacy</a> |
<a class="Link" href="https://www.nvidia.com/en-us/preferences/start/" target="_blank" rel="noopener"
data-cms-ai="0">Do Not Sell or Share My Data</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/terms-of-service/" target="_blank"
rel="noopener" data-cms-ai="0">Terms of Service</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/accessibility/" target="_blank" rel="noopener"
data-cms-ai="0">Accessibility</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/company-policies/" target="_blank"
rel="noopener" data-cms-ai="0">Corporate Policies</a> |
<a class="Link" href="https://www.nvidia.com/en-us/product-security/" target="_blank" rel="noopener"
data-cms-ai="0">Product Security</a> |
<a class="Link" href="https://www.nvidia.com/en-us/contact/" target="_blank" rel="noopener"
data-cms-ai="0">Contact</a>
</p>
</div>
</div>
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink