kanshan/TensorRT-LLMs
1
0
Fork 0
mirror of https://github.com/NVIDIA/TensorRT-LLM.git synced 2026-01-14 06:27:45 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
85f78df69c
TensorRT-LLMs/_cpp_gen/executor.html
nv-guomingz 85f78df69c
Update gh-pages for windows part doc. (#1979) 
Co-authored-by: Guoming Zhang <37257613+nv-guomingz@users.noreply.github.com>
2024-07-18 11:18:09 +08:00

4770 lines
1.0 MiB
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Executor &mdash; tensorrt_llm documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=80d5e7a1" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=19f00094" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="../_static/jquery.js?v=5d32c60e"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=9a2dae69"></script>
<script src="../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Runtime" href="runtime.html" />
<link rel="prev" title="Memory Usage of TensorRT-LLM" href="../reference/memory.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
tensorrt_llm
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Getting Started</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../quick-start-guide.html">Quick Start Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">Release Notes</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Installation</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../installation/linux.html">Installing on Linux</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/build-from-source-linux.html">Building from Source Code on Linux</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/windows.html">Installing on Windows</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation/build-from-source-windows.html">Building from Source Code on Windows</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Architecture</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../architecture/overview.html">TensorRT-LLM Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html">Model Definition</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html#compilation">Compilation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html#runtime">Runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/core-concepts.html#multi-gpu-and-multi-node-support">Multi-GPU and Multi-Node Support</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/checkpoint.html">TensorRT-LLM Checkpoint</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/workflow.html">TensorRT-LLM Build Workflow</a></li>
<li class="toctree-l1"><a class="reference internal" href="../architecture/add-model.html">Adding a Model</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Advanced</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../advanced/gpt-attention.html">Multi-Head, Multi-Query, and Group-Query Attention</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/gpt-runtime.html">C++ GPT Runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/graph-rewriting.html">Graph Rewriting Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/batch-manager.html">The Batch Manager in TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/inference-request.html">Inference Request</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/inference-request.html#responses">Responses</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/lora.html">Run gpt-2b + LoRA using GptManager / cpp runtime</a></li>
<li class="toctree-l1"><a class="reference internal" href="../advanced/expert-parallelism.html">Expert Parallelism in TensorRT-LLM</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Performance</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-overview.html">Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-best-practices.html">Best Practices for Tuning the Performance of TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../performance/perf-analysis.html">Performance Analysis</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Reference</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../reference/troubleshooting.html">Troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/support-matrix.html">Support Matrix</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/precision.html">Numerical Precision</a></li>
<li class="toctree-l1"><a class="reference internal" href="../reference/memory.html">Memory Usage of TensorRT-LLM</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">C++ API</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Executor</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#executor-h">executor.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv412tensorrt_llm"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::mpi</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#serialization-h">serialization.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13SerializationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSamplingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeOutputConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeExternalDraftTokensConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializePromptTuningConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeLoraConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeRequest()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeTensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeResult()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeResponse()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeResponses()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeKvCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeSchedulerConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeParallelConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializePeftCacheConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeOrchestratorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDecodingMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeLookaheadDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeDecodingConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeExecutorConfig()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeKvCacheStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeStaticBatchingStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeInflightBatchingStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeIterationStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeIterationStats()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serialize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::serializedSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeString()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeBool()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Serialization::deserializeModelType()</span></code></a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#tensor-h">tensor.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Base</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::DimType64</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Shape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Shape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Shape::Shape()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::CudaStreamPtr</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToCpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToPooledPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToManaged()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyToGpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorD0Ev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::~Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERR6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7getDataEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getData()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getDataEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getData()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor11getDataTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getDataType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getMemoryType()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor8getShapeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getShape()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getSizeInBytes()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::setZero()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::setFrom()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6TensorcvbEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator</span> <span class="pre">bool()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6TensoreqERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6TensorneERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::operator!=()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::cpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3cpuE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::cpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor6pinnedE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pooledPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::pooledPinned()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::managed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor7managedE6Tensor5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::managed()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::gpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3gpuE6Tensor13CudaStreamPtr5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::gpu()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::of()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::of()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::of()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Impl</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::Tensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::copyTo()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor7mTensorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::mTensor</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeE8DataTypev"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::getRuntimeType()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::toITensor()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::Tensor::ofITensor()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detailE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail::DimType64</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9toITensorERK6Tensor"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail::toITensor()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::detail::ofITensor()</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::runtime</span></code></a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#types-h">types.h</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE"><code class="docutils literal notranslate"><span class="pre">TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE"><code class="docutils literal notranslate"><span class="pre">TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9TensorPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TensorPtr</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::SizeType32</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::FloatType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TokenIdType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::VecTokens</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BeamTokens</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IdType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RandomSeedType</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::VecLogProbs</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StreamPtr</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessor</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorMap</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::LogitsPostProcessorBatched</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MedusaChoices</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kBOOL</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kUINT8</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kINT8</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kINT32</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kINT64</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBF16E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kBF16</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType4kFP8E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kFP8</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP16E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kFP16</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP32E"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kFP32</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType8kUNKNOWNE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DataType::kUNKNOWN</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kCPUE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kCPU</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kCPU_PINNED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kGPUE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kGPU</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kUVME"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kUVM</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryType8kUNKNOWNE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::MemoryType::kUNKNOWN</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType::kDECODER_ONLY</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType::kENCODER_ONLY</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ModelType::kENCODER_DECODER</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BatchingType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingType7kSTATICE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BatchingType::kSTATIC</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::BatchingType::kINFLIGHT</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy::kMAX_UTILIZATION</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextChunkingPolicy</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextChunkingPolicy::kFIRST_COME_FIRST_SERVED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::ContextChunkingPolicy::kEQUAL_PROGRESS</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationType</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationType::kMPI</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationMode</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationMode::kLEADER</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::CommunicationMode::kORCHESTRATOR</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage7kQUEUEDE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kQUEUED</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kENCODER_IN_PROGRESS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kCONTEXT_IN_PROGRESS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kGENERATION_IN_PROGRESS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStage::kGENERATION_COMPLETE</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::operator&lt;&lt;()</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0_bEN12tensorrt_llm8executor10TypeTraitsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIfE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsI4halfEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19PhonyNameDueToError5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::PhonyNameDueToError::value</span></code></a></li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIbEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIbE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::TypeTraits::value</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::maxNumBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::freeNumBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::usedNumBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::tokensPerBlock</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::allocTotalBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::allocNewBlocks</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::KvCacheStats::reusedBlocks</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numScheduledRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numContextRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numCtxTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::numGenTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::StaticBatchingStats::emptyGenSlots</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numScheduledRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numContextRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numGenRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numPausedRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::numCtxTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::microBatchId</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::InflightBatchingStats::avgNumDecodedTokensPerIter</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats9timestampE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::timestamp</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats4iterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::iter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats13iterLatencyMSE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::iterLatencyMS</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numActiveRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::numActiveRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::numQueuedRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::maxNumActiveRequests</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11gpuMemUsageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::gpuMemUsage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11cpuMemUsageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::cpuMemUsage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::pinnedMemUsage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats12kvCacheStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::kvCacheStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::crossKvCacheStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::staticBatchingStats</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::IterationStats::inflightBatchingStats</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats2idE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::id</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats5stageE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::stage</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::contextPrefillPosition</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::numGeneratedTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::avgNumDecodedTokensPerIter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats9scheduledE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::scheduled</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStats6pausedE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStats::paused</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStatsPerIteration</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration4iterE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStatsPerIteration::iter</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::RequestStatsPerIteration::requestStats</span></code></a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode</span></code></a><ul>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::UnderlyingType</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14useTemperatureEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useOccurrencePenalties()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::usePresencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useFrequencyPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useMinLengthEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useMinLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useBanTokensEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useBanTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11useBanWordsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useBanWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useStopWordsEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useStopWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useMaxLengthStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::useExplicitEosStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isAutoEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isAuto()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopKEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopKorTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isTopKandTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isBeamSearch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8isMedusaEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isMedusa()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isLookahead()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isExplicitDraftTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseTemperature()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUsePresencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseFrequencyPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseRepetitionPenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseMinLength()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseOccurrencePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUsePenalty()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseBanWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseNoRepeatNgramSize()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseBanTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseStopWords()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseMaxLengthStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseExplicitEosStop()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::isUseStopCriteria()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::operator==()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::DecodingMode()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8getStateEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::getState()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4AutoEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Auto()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopKEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::TopK()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::TopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8TopKTopPEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::TopKTopP()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10BeamSearchEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::BeamSearch()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6MedusaEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Medusa()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9LookaheadEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::Lookahead()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::ExplicitDraftTokens()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::anyBitSet()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::allBitSet()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::setBitTo()</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6mStateE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::mState</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseRepetitionPenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseFrequencyPenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUsePresencePenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseTemperature</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseMinLength</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseBanWords</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseStopWords</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseMaxLengthStop</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseExplicitEosStop</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseNoRepeatNgramSize</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kStandardStopCriteria</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseOccurrencePenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUsePenalties</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kUseBanTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kNumFlags</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kAutoE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kAuto</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kTopK</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kTopP</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11kBeamSearchE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kBeamSearch</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode7kMedusaE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kMedusa</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10kLookaheadE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kLookahead</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kExplicitDraftTokens</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kTopKTopPE"><code class="docutils literal notranslate"><span class="pre">tensorrt_llm::executor::DecodingMode::kTopKTopP</span></code></a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="runtime.html">Runtime</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Blogs</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../blogs/H100vsA100.html">H100 has 4.6x A100 Performance in TensorRT-LLM, achieving 10,000 tok/s at 100ms to first token</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/H200launch.html">H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/Falcon180B-H200.html">Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/quantization-in-TRT-LLM.html">Speed up inference with SOTA quantization techniques in TRT-LLM</a></li>
<li class="toctree-l1"><a class="reference internal" href="../blogs/XQA-kernel.html">New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">tensorrt_llm</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active">Executor</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/_cpp_gen/executor.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="executor">
<h1>Executor<a class="headerlink" href="#executor" title="Link to this heading"></a></h1>
<section id="executor-h">
<h2>executor.h<a class="headerlink" href="#executor-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv412tensorrt_llm">
<span id="_CPPv312tensorrt_llm"></span><span id="_CPPv212tensorrt_llm"></span><span id="tensorrt_llm"></span><span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><a class="headerlink" href="#_CPPv412tensorrt_llm" title="Link to this definition"></a><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executorE">
<span id="_CPPv3N12tensorrt_llm8executorE"></span><span id="_CPPv2N12tensorrt_llm8executorE"></span><span id="tensorrt_llm::executor"></span><span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executorE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-variables">Variables</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor30kDefaultIterStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor30kDefaultIterStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor30kDefaultIterStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::kDefaultIterStatsMaxIterations__SizeType32C"></span><span class="target" id="executor_8h_1a705863a87f62b548dca56404d20e2f22"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultIterStatsMaxIterations</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1000</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor30kDefaultIterStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor33kDefaultRequestStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor33kDefaultRequestStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor33kDefaultRequestStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::kDefaultRequestStatsMaxIterations__SizeType32C"></span><span class="target" id="executor_8h_1a1bb94f27129131a88ea89a5380021aa4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDefaultRequestStatsMaxIterations</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor33kDefaultRequestStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfigE"></span><span id="tensorrt_llm::executor::SamplingConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Sampling configuration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::SamplingConfig__SizeType32.std::optional:SizeType32:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:TokenIdType:CR.std::optional:FloatType:CR.std::optional:RandomSeedType:CR.std::optional:FloatType:CR.std::optional:SizeType32:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:FloatType:CR.std::optional:SizeType32:CR.std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ad0a2ce7abec5c8dd95830d94b09850cd"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SamplingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topK</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topP</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPMin</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPResetIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPDecay</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">randomSeed</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">temperature</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minLength</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">beamSearchDiversityRate</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">repetitionPenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">presencePenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">frequencyPenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lengthPenalty</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">earlyStopping</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14SamplingConfigE10SizeType32RKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI11TokenIdTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI14RandomSeedTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI9FloatTypeEERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd><p>Constructor for <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1SamplingConfig"><span class="std std-ref">SamplingConfig</span></a> See description of parameters below. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig"></span><span id="tensorrt_llm::executor::SamplingConfig::eq-operator__SamplingConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a08919f8eb49492f6e0f23f20b2ff0555"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfigeqERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getBeamWidthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9f3845e5dabe1f14aa2130c7adc8dc9d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBeamWidth</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getBeamWidthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopKEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig7getTopKEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig7getTopKEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopKC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af6f531e3c3d14651899a2cbec6dce470"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopK</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopKEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig7getTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig7getTopPEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ad35d070b37dd9352a87b09e5a74000f1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig7getTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPMinC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5491c09e86ed77db5b01af0db541dc9e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopPMin</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig10getTopPMinEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPResetIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a0d7d9a49e9fde68f22dbb15fff438893"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopPResetIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig15getTopPResetIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTopPDecayC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a2e5aaf01096ce76e81e7d7063f355942"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTopPDecay</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getTopPDecayEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getRandomSeedC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5397f4d76e598c910f0d89532864bc78"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRandomSeed</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig13getRandomSeedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getTemperatureC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a368dcbf925a19227614b1bcc023d6fd8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTemperature</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig14getTemperatureEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getMinLengthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a12f414ad240126cd2468753d0527f663"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMinLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig12getMinLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getBeamSearchDiversityRateC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aeafb5afe911654f9b167fb3c5294fb05"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBeamSearchDiversityRate</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig26getBeamSearchDiversityRateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getRepetitionPenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aac623697b4549a0fd2a1e4885b7c1d81"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getRepetitionPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getPresencePenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a407e6c1fff816ef21e0338e578962c97"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPresencePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig18getPresencePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getFrequencyPenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a805d401be8f7059fc6ae7c7172b129b1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig19getFrequencyPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getLengthPenaltyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a896a92d1e3252ed7eb3079e885eefbe8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLengthPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getLengthPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getEarlyStoppingC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5912f74caf40ae8fa212cf014ddd740d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEarlyStopping</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig16getEarlyStoppingEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv"></span><span id="tensorrt_llm::executor::SamplingConfig::getNoRepeatNgramSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a37c26daaec9e0d50ddbee109f2154138"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14SamplingConfig20getNoRepeatNgramSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32"></span><span id="tensorrt_llm::executor::SamplingConfig::setBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9d408d845ffd468e5c77a12644580acc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBeamWidth</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setBeamWidthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopK__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5461168ba842d21d25c4c584d9f9d023"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopK</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topK</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopKERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopP__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a913f54f64881543acd7709ed99da5085"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopP</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topP</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig7setTopPERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopPMin__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aa1789d40088cb15c5a07398607f2dd76"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopPMin</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPMin</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10setTopPMinERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopPResetIds__std::optional:TokenIdType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab4926172b96ae86559d6247ba0b55b51"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopPResetIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPResetIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig15setTopPResetIdsERKNSt8optionalI11TokenIdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTopPDecay__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab897fc960092c5e35a20e5eee4849d2d"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTopPDecay</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPDecay</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setTopPDecayERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setRandomSeed__std::optional:RandomSeedType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a7cc025262a443afb49884619e7d0dbd4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRandomSeed</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">randomSeed</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig13setRandomSeedERKNSt8optionalI14RandomSeedTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setTemperature__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a22735fef3e4a93fb9d6f05e2405c8fc0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setTemperature</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">temperature</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14setTemperatureERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setMinLength__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a8cd60f4e1ba6a48483ae02608eb31385"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMinLength</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12setMinLengthERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setBeamSearchDiversityRate__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab22c4602e2c4ad41a8b5896d914d9a0c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBeamSearchDiversityRate</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">beamSearchDiversityRate</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig26setBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setRepetitionPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac58bfcdc98186d18963554bb4b6c01f8"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">repetitionPenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20setRepetitionPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setPresencePenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1abe8dd822d08b0210e9f1fa3b36490fdf"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPresencePenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">presencePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18setPresencePenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setFrequencyPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a468c9e296264e676bd40c884c85bc4da"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">frequencyPenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig19setFrequencyPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::setLengthPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a9ea9013d86101dcdfb9d15a2b25998f5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLengthPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lengthPenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16setLengthPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setEarlyStopping__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a20097f9517e3f46e6584addf31a8fb3f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEarlyStopping</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">earlyStopping</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16setEarlyStoppingERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::setNoRepeatNgramSize__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1add1f574a466f0c5a89933fe6f0d3dc13"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig20setNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE"></span><span id="tensorrt_llm::executor::SamplingConfig::mBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab75403888fa3c8d301912dbc350c35fb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBeamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mBeamWidthE" title="Link to this definition"></a><br /></dt>
<dd><p>The beam width. Default is 1 which disables beam search. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopKE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig5mTopKE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig5mTopKE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopK__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac9850671f2d8eed17ecd6e8e4845f401"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopK</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopKE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls number of logits to sample from. Default is 0 (all logits). </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopPE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig5mTopPE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig5mTopPE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopP__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac442a2bcecf64be99729c882c9aaf09b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopP</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig5mTopPE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the top-P probability to sample from. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig8mTopPMinE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig8mTopPMinE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig8mTopPMinE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopPMin__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac0692f29ae762728c4a7d1e438dad91e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopPMin</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig8mTopPMinE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls decay in the top-P algorithm. topPMin is lower-bound. Default is 1.e-6. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopPResetIds__std::optional:TokenIdType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab0c410d2557fdf327ede1d67c292a7a2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopPResetIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig13mTopPResetIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls decay in the top-P algorithm. Indicates where to reset the decay. Default is 1. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTopPDecay__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1af161a37e4aa10723d8ae5627943b4e03"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTopPDecay</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mTopPDecayE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls decay in the top-P algorithm. The decay value. Default is 1.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig11mRandomSeedE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig11mRandomSeedE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig11mRandomSeedE"></span><span id="tensorrt_llm::executor::SamplingConfig::mRandomSeed__std::optional:RandomSeedType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a6ff437179e5deeae113fd53796566c73"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="tensorrt_llm::executor::RandomSeedType"><span class="n"><span class="pre">RandomSeedType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRandomSeed</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig11mRandomSeedE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the random seed used by the random number generator in sampling. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12mTemperatureE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12mTemperatureE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12mTemperatureE"></span><span id="tensorrt_llm::executor::SamplingConfig::mTemperature__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ab5b9a3e150bdc536de54725c9210281e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTemperature</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12mTemperatureE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the modulation of logits when sampling new tokens. It can have values &gt; 0.f. Default is 1.0f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig10mMinLengthE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig10mMinLengthE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig10mMinLengthE"></span><span id="tensorrt_llm::executor::SamplingConfig::mMinLength__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a93e30780eeb90cf0ef2573bc4f802b39"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMinLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig10mMinLengthE" title="Link to this definition"></a><br /></dt>
<dd><p>Lower bound on the number of tokens to generate. Values &lt; 1 have no effect. Default is 1. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE"></span><span id="tensorrt_llm::executor::SamplingConfig::mBeamSearchDiversityRate__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5d7fb8264ede8714d68ee323af7ce58f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBeamSearchDiversityRate</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig24mBeamSearchDiversityRateE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the diversity in beam search. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mRepetitionPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1afc6184e856da9b06c54fd9093ca517bc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRepetitionPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18mRepetitionPenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Used to penalize tokens based on how often they appear in the sequence. It can have any value &gt; 0.f. Values &lt; 1.f encourages repetition, values &gt; 1.f discourages it. Default is 1.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mPresencePenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1acee024920425ee593d16ac95113d5e2d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPresencePenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16mPresencePenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Used to penalize tokens already present in the sequence (irrespective of the number of appearances). It can have any values. Values &lt; 0.f encourage repetition, values &gt; 0.f discourage it. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mFrequencyPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a537a6f894e3c3093957c46774b23bb38"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFrequencyPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig17mFrequencyPenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Used to penalize tokens already present in the sequence (dependent on the number of appearances). It can have any values. Values &lt; 0.f encourage repetition, values &gt; 0.f discourage it. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE"></span><span id="tensorrt_llm::executor::SamplingConfig::mLengthPenalty__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a131e7f83f8b2a827761284f1bca2f934"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLengthPenalty</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14mLengthPenaltyE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls how to penalize longer sequences in beam search. Default is 0.f. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE"></span><span id="tensorrt_llm::executor::SamplingConfig::mEarlyStopping__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a5bf3c3f8361b9ffd284f386ccd69eab0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEarlyStopping</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14mEarlyStoppingE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls whether the generation process finishes once beamWidth sentences are generated (ends with end_token) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE"></span><span id="tensorrt_llm::executor::SamplingConfig::mNoRepeatNgramSize__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a782d671e6803a1cd7c3116004082b42e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNoRepeatNgramSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig18mNoRepeatNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls how many repeat ngram size are acceptable. Default is 1 &lt;&lt; 30. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32"></span><span id="tensorrt_llm::executor::SamplingConfig::checkBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a766b90d2d0f211808369d11906e561f6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">checkBeamWidth</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">beamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkBeamWidthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopK__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aeda79a4db482cf83abc965736f11af1a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopK</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topK</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopKERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopP__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1acbd6ffb8df5eae6b19dcb4f9fc559231"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopP</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topP</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig9checkTopPERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopPMin__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1ac1a038c4ef4208385416e8c0d8dc5d41"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopPMin</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPMin</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig12checkTopPMinERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopPResetIds__std::optional:TokenIdType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a7fc3eeb83c34dbbf1ac27bfe39f519f8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopPResetIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPResetIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig17checkTopPResetIdsERKNSt8optionalI11TokenIdTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTopPDecay__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a924fb8d5567785e6a7066c106b7947f0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTopPDecay</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">topPDecay</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkTopPDecayERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkTemperature__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1aad7365c70021e15f8820d428e9583e96"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkTemperature</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">temperature</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig16checkTemperatureERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkRepetitionPenalty__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a2a92b48cbe1ed7b0f42bfcd21c677a84"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">penalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkRepetitionPenaltyERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkMinLengthERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig14checkMinLengthERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig14checkMinLengthERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkMinLength__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a119a591373b22edef454f3aed88cee87"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkMinLength</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">minLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig14checkMinLengthERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkNoRepeatNgramSize__std::optional:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a69903b86de68c646c159ce70e1e63343"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig22checkNoRepeatNgramSizeERKNSt8optionalI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::SamplingConfig::checkBeamSearchDiversityRate__std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SamplingConfig_1a6a12e04f21c9b25067b2ae68c329a569"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">checkBeamSearchDiversityRate</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">beamSearchDiversityRate</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfig28checkBeamSearchDiversityRateERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfigE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfigE"></span><span id="tensorrt_llm::executor::OutputConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OutputConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration that controls the outputs of a <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a>. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbb">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbb"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbb"></span><span id="tensorrt_llm::executor::OutputConfig::OutputConfig__b.b.b.b.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a5fdfba54f2dabcfd47a5e5212206974b"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OutputConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnLogProbs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnContextLogits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnGenerationLogits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">excludeInputFromOutput</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">returnEncoderOutput</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig12OutputConfigEbbbbb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig14returnLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig14returnLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig14returnLogProbsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a1f23a994b65629c8beee0bbb79f71ee7"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig14returnLogProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain log probabilities. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnContextLogits__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a4419de36cc2f99f8c4ce8911ee0dadf5"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnContextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig19returnContextLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain the context logits. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE"></span><span id="tensorrt_llm::executor::OutputConfig::returnGenerationLogits__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a1d8dda4081c539f18f32af0be70c0f17"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnGenerationLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig22returnGenerationLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain the generation logits. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE"></span><span id="tensorrt_llm::executor::OutputConfig::excludeInputFromOutput__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1ab38293c37f6f6b60e3e39f68f18f7b90"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">excludeInputFromOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig22excludeInputFromOutputE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if output tokens in <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should include the input tokens. Default is false. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE">
<span id="_CPPv3N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE"></span><span id="_CPPv2N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE"></span><span id="tensorrt_llm::executor::OutputConfig::returnEncoderOutput__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OutputConfig_1a11049bc5a1b255c703533515539ec66a"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">returnEncoderOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12OutputConfig19returnEncoderOutputE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if <a class="reference internal" href="#structtensorrt__llm_1_1executor_1_1Result"><span class="std std-ref">Result</span></a> should contain encoder output hidden states (for encoder-only and encoder-decoder models). Default is false. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfigE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for speculative decoding with external draft tokens. Allows to include draft tokens, draft logits and specify acceptance threshold. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::ExternalDraftTokensConfig__VecTokens.std::optional:Tensor:.std::optional:FloatType:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1aeceae59dc463c61c0bdd7eaacad9f7da"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">tokens</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logits</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">acceptanceThreshold</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig25ExternalDraftTokensConfigE9VecTokensNSt8optionalI6TensorEERKNSt8optionalI9FloatTypeEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1ad99090a218772e934a91558571ea4f67"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getLogitsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a6b92e95a2a620512ae57000e9492d1bc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogits</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig9getLogitsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv">
<span id="_CPPv3NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::getAcceptanceThresholdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a3f5a3ac2d281ee7c8766b06dcb68b6a2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getAcceptanceThreshold</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor25ExternalDraftTokensConfig22getAcceptanceThresholdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mTokens__VecTokens"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1a648382f7b513a2fb82330540687833e3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The draft tokens. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mLogits__std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1add7e98a70f09f15c3390a39a7673b5a5"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig7mLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>The draft logits. Expected shape: [num_draft_tokens, vocab_size]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE">
<span id="_CPPv3N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE"></span><span id="_CPPv2N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE"></span><span id="tensorrt_llm::executor::ExternalDraftTokensConfig::mAcceptanceThreshold__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExternalDraftTokensConfig_1abcc619153ad5a5e2c5bf08940eda9064"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mAcceptanceThreshold</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfig20mAcceptanceThresholdE" title="Link to this definition"></a><br /></dt>
<dd><p>The acceptance threshold. Must be &gt; 0.f and &lt;= 1.f. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfigE"></span><span id="tensorrt_llm::executor::PromptTuningConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for prompt tuning. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6Tensor"></span><span id="tensorrt_llm::executor::PromptTuningConfig::PromptTuningConfig__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1a205fc6a8d9680944df5e3afb4b899ef5"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PromptTuningConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">embeddingTable</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig18PromptTuningConfigE6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv">
<span id="_CPPv3NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv"></span><span id="tensorrt_llm::executor::PromptTuningConfig::getEmbeddingTableC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1af8534fb4246a662291be6f7e17655ca0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEmbeddingTable</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18PromptTuningConfig17getEmbeddingTableEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE">
<span id="_CPPv3N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE"></span><span id="_CPPv2N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE"></span><span id="tensorrt_llm::executor::PromptTuningConfig::mEmbeddingTable__Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PromptTuningConfig_1a4f6be4050b7b94a5950234e70b9df919"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEmbeddingTable</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfig15mEmbeddingTableE" title="Link to this definition"></a><br /></dt>
<dd><p>The prompt embedding table. Expected shape: [task vocab_size, hidden_size]. Data type must match model weights. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfigE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfigE"></span><span id="tensorrt_llm::executor::LoraConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LoraConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration for LoRA. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE"></span><span id="tensorrt_llm::executor::LoraConfig::LoraConfig__IdType.std::optional:Tensor:.std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a2a12dcb55bc38dd108c977c7b2fb4a55"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LoraConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">taskId</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">weights</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">config</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig10LoraConfigE6IdTypeNSt8optionalI6TensorEENSt8optionalI6TensorEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv"></span><span id="tensorrt_llm::executor::LoraConfig::getTaskIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1ad1d8cbbfcbd2046bf1b73ce608582c1d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getTaskId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig9getTaskIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor10LoraConfig10getWeightsEv">
<span id="_CPPv3NK12tensorrt_llm8executor10LoraConfig10getWeightsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor10LoraConfig10getWeightsEv"></span><span id="tensorrt_llm::executor::LoraConfig::getWeightsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a625979d46bc9a9310c486bf6b54c0969"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWeights</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig10getWeightsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor10LoraConfig9getConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor10LoraConfig9getConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor10LoraConfig9getConfigEv"></span><span id="tensorrt_llm::executor::LoraConfig::getConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a521f867bec63e16f10c24fe8d6953f9f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor10LoraConfig9getConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig7mTaskIdE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig7mTaskIdE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig7mTaskIdE"></span><span id="tensorrt_llm::executor::LoraConfig::mTaskId__IdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a863ffa00f268192c572d4e3f0737d535"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTaskId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig7mTaskIdE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora task id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig8mWeightsE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig8mWeightsE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig8mWeightsE"></span><span id="tensorrt_llm::executor::LoraConfig::mWeights__std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1ae6941906a80853143186df62f10d8d95"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWeights</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig8mWeightsE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora weights. See TRT-LLM documentation for expected shapes and types. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10LoraConfig7mConfigE">
<span id="_CPPv3N12tensorrt_llm8executor10LoraConfig7mConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor10LoraConfig7mConfigE"></span><span id="tensorrt_llm::executor::LoraConfig::mConfig__std::optional:Tensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1LoraConfig_1a70553cf896bdbb10718070e7da355bbc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10LoraConfig7mConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The Lora configuration. See TRT-LLM documentation for detailed description of the config tensor. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestE">
<span id="_CPPv3N12tensorrt_llm8executor7RequestE"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestE"></span><span id="tensorrt_llm::executor::Request"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>A class that holds information about the request. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI10LoraConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI10LoraConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI10LoraConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEE"></span><span id="tensorrt_llm::executor::Request::Request__VecTokens.SizeType32.b.SamplingConfigCR.OutputConfigCR.std::optional:SizeType32:CR.std::optional:SizeType32:CR.std::optional:std::list:VecTokens::.std::optional:std::list:VecTokens::.std::optional:Tensor:.std::optional:ExternalDraftTokensConfig:.std::optional:PromptTuningConfig:.std::optional:LoraConfig:.std::optional:ss:.std::optional:VecTokens:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae887e8b2b27093a73dc90a3b669bc327"></span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">inputTokenIds</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxNewTokens</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">streaming</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">samplingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">endId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">padId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">badWords</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stopWords</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">embeddingBias</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">externalDraftTokensConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">pTuningConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">loraConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsPostProcessorName</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encoderInputTokenIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestE9VecTokens10SizeType32bRK14SamplingConfigRK12OutputConfigRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EENSt8optionalINSt4listI9VecTokensEEEENSt8optionalINSt4listI9VecTokensEEEENSt8optionalI6TensorEENSt8optionalI25ExternalDraftTokensConfigEENSt8optionalI18PromptTuningConfigEENSt8optionalI10LoraConfigEENSt8optionalINSt6stringEEENSt8optionalI9VecTokensEE" title="Link to this definition"></a><br /></dt>
<dd><p>The <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> constructor. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>inputTokenIds</strong> – The input token ids </p></li>
<li><p><strong>maxNewTokens</strong> – The maximum number of tokens to generate </p></li>
<li><p><strong>streaming</strong> – Indicates if the responses should be streamed or not. Default is false. </p></li>
<li><p><strong>samplingConfig</strong> – The sampling configuration </p></li>
<li><p><strong>outputConfig</strong> – The output configuration </p></li>
<li><p><strong>endId</strong> – The end token id </p></li>
<li><p><strong>padId</strong> – The pad token id </p></li>
<li><p><strong>badWords</strong> – A list of bad words tokens. Each “word” can be composed of multiple tokens </p></li>
<li><p><strong>stopWords</strong> – A list of stop words tokens. Each “word” can be composed of multiple tokens </p></li>
<li><p><strong>embeddingBias</strong> – The embedding bias tensor. Expected type is kFP32 and shape is [vocab_size] </p></li>
<li><p><strong>externalDraftTokensConfig</strong> – The speculative decoding configuration </p></li>
<li><p><strong>pTuningConfig</strong> – The prompt tuning configuration </p></li>
<li><p><strong>loraConfig</strong> – The LoRA configuration </p></li>
<li><p><strong>logitsPostProcessorName</strong> – The logits postprocessor name. Must correspond to one of the logits postprocessor name provided to the <a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1ExecutorConfig"><span class="std std-ref">ExecutorConfig</span></a>. </p></li>
<li><p><strong>encoderInputTokenIds</strong> – The encoder input token ids for encoder-decoder models, or encoder-only models </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor7Request7RequestERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7Request7RequestERK7Request"></span><span id="tensorrt_llm::executor::Request::Request__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ac1ba647ff89d0de162cb6b7e8a3b3218"></span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request" title="tensorrt_llm::executor::Request::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERK7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request">
<span id="_CPPv3N12tensorrt_llm8executor7Request7RequestERR7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7Request7RequestERR7Request"></span><span id="tensorrt_llm::executor::Request::Request__RequestRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aea6f0d55747d21a96b77e164de0dcacb"></span><span class="sig-name descname"><span class="n"><span class="pre">Request</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request" title="tensorrt_llm::executor::Request::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request7RequestERR7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestaSERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor7RequestaSERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestaSERK7Request"></span><span id="tensorrt_llm::executor::Request::assign-operator__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ab9b7fb6eff1f0fad0a70ea63b8765d59"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestaSERK7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestaSERR7Request">
<span id="_CPPv3N12tensorrt_llm8executor7RequestaSERR7Request"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestaSERR7Request"></span><span id="tensorrt_llm::executor::Request::assign-operator__RequestRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad93719fa8239c07341d613b6b1fc7c51"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestaSERR7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7RequestD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor7RequestD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor7RequestD0Ev"></span><span id="tensorrt_llm::executor::Request::~Request"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad615bd9f0869e60c95bb9dcb01ed8ed6"></span><span class="sig-name descname"><span class="n"><span class="pre">~Request</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7RequestD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request16getInputTokenIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request16getInputTokenIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request16getInputTokenIdsEv"></span><span id="tensorrt_llm::executor::Request::getInputTokenIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae5023d8d2d8cfa9bd3e3a16c76cc6819"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getInputTokenIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request16getInputTokenIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request15getMaxNewTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request15getMaxNewTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request15getMaxNewTokensEv"></span><span id="tensorrt_llm::executor::Request::getMaxNewTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a357aeecb46381de018ddd3836b1f23d3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxNewTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request15getMaxNewTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request12getStreamingEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request12getStreamingEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request12getStreamingEv"></span><span id="tensorrt_llm::executor::Request::getStreamingC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a04fe14eab1bf2031220dfce904f6e736"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getStreaming</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request12getStreamingEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request17getSamplingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request17getSamplingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request17getSamplingConfigEv"></span><span id="tensorrt_llm::executor::Request::getSamplingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae89bc1ae8e0a108e2f74e43470631e4a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSamplingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request17getSamplingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request15getOutputConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request15getOutputConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request15getOutputConfigEv"></span><span id="tensorrt_llm::executor::Request::getOutputConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a4fe15a7d4be498a08b8dcbe999a4241d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOutputConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request15getOutputConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request8getEndIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request8getEndIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request8getEndIdEv"></span><span id="tensorrt_llm::executor::Request::getEndIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a75d6f59cb5b386468a9164c47c3bd476"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEndId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request8getEndIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request8getPadIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request8getPadIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request8getPadIdEv"></span><span id="tensorrt_llm::executor::Request::getPadIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aec4d6a17d1a5a4dfb13def37154f5aa3"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPadId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request8getPadIdEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request11getBadWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request11getBadWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request11getBadWordsEv"></span><span id="tensorrt_llm::executor::Request::getBadWordsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1adf65522c1e4c5578e4ae36fefbda1e85"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBadWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request11getBadWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request12getStopWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request12getStopWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request12getStopWordsEv"></span><span id="tensorrt_llm::executor::Request::getStopWordsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a52d0bf6d311bc908e8b44d5f327183fe"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getStopWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request12getStopWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv"></span><span id="tensorrt_llm::executor::Request::getEmbeddingBiasC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a1bda1696484106868ec92e5ad476ea36"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEmbeddingBias</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request16getEmbeddingBiasEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv"></span><span id="tensorrt_llm::executor::Request::getExternalDraftTokensConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aa5cf661686e4c873f5055bb2f313c1a2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request28getExternalDraftTokensConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv"></span><span id="tensorrt_llm::executor::Request::getPromptTuningConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a03d917958773e5b2adc4913a8ec1f17c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPromptTuningConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request21getPromptTuningConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request13getLoraConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request13getLoraConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request13getLoraConfigEv"></span><span id="tensorrt_llm::executor::Request::getLoraConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a9c92065c4255500f50b3298664604b26"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLoraConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request13getLoraConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv"></span><span id="tensorrt_llm::executor::Request::getLogitsPostProcessorNameC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1aa2d7390d628610c460c0e1af413c7ee2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogitsPostProcessorName</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request26getLogitsPostProcessorNameEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv"></span><span id="tensorrt_llm::executor::Request::getEncoderInputTokenIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a71c376a85795efba4c238b33cb5168f7"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEncoderInputTokenIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor7Request23getEncoderInputTokenIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request12setStreamingEb">
<span id="_CPPv3N12tensorrt_llm8executor7Request12setStreamingEb"></span><span id="_CPPv2N12tensorrt_llm8executor7Request12setStreamingEb"></span><span id="tensorrt_llm::executor::Request::setStreaming__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae68a921ab34759ffc097e342253d52f9"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setStreaming</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">streaming</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request12setStreamingEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig"></span><span id="tensorrt_llm::executor::Request::setSamplingConfig__SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1abe649c4845388cea78bf22c291538341"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSamplingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request17setSamplingConfigERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig"></span><span id="tensorrt_llm::executor::Request::setOutputConfig__OutputConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a73a7ec9973c9a76faf89738aedaa69f2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOutputConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">outputConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request15setOutputConfigERK12OutputConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request8setEndIdE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor7Request8setEndIdE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor7Request8setEndIdE10SizeType32"></span><span id="tensorrt_llm::executor::Request::setEndId__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a76a2cc1cc476cb0a050c24a73a6e5cbc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEndId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">endId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request8setEndIdE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request8setPadIdE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor7Request8setPadIdE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor7Request8setPadIdE10SizeType32"></span><span id="tensorrt_llm::executor::Request::setPadId__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a8f1d378a89c237f2c79f5510d285a32e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPadId</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">padId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request8setPadIdE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE"></span><span id="tensorrt_llm::executor::Request::setBadWords__std::list:VecTokens:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ae2c911bc838c38ff9c0955063d532b5b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBadWords</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">badWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request11setBadWordsERKNSt4listI9VecTokensEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE">
<span id="_CPPv3N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE"></span><span id="tensorrt_llm::executor::Request::setStopWords__std::list:VecTokens:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ac3fab641b66477d4c11446a7a09f9f72"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setStopWords</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">stopWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request12setStopWordsERKNSt4listI9VecTokensEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor"></span><span id="tensorrt_llm::executor::Request::setEmbeddingBias__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a3184ce8fb36c13840dec0e219056e3e1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEmbeddingBias</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">embeddingBias</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request16setEmbeddingBiasERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig"></span><span id="tensorrt_llm::executor::Request::setExternalDraftTokensConfig__ExternalDraftTokensConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a311a94236786522ab77f47fe38bd0402"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">externalDraftTokensConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request28setExternalDraftTokensConfigERK25ExternalDraftTokensConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig"></span><span id="tensorrt_llm::executor::Request::setPromptTuningConfig__PromptTuningConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ad01f7cf40b183ad8b7c77ee0c03d8c51"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPromptTuningConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">pTuningConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request21setPromptTuningConfigERK18PromptTuningConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig">
<span id="_CPPv3N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig"></span><span id="_CPPv2N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig"></span><span id="tensorrt_llm::executor::Request::setLoraConfig__LoraConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1ada0604e9f9989519d8595e5b0777d875"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLoraConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">loraConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request13setLoraConfigERK10LoraConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE"></span><span id="tensorrt_llm::executor::Request::setLogitsPostProcessorName__ssCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a26704f9863977008bd456b1aa49e3e55"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLogitsPostProcessorName</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logitsPostProcessorName</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request26setLogitsPostProcessorNameERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens">
<span id="_CPPv3N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens"></span><span id="_CPPv2N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens"></span><span id="tensorrt_llm::executor::Request::setEncoderInputTokenIds__VecTokensCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1adf41e2b961d2bd01a43ea7ef17d19a01"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEncoderInputTokenIds</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderInputTokenIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request23setEncoderInputTokenIdsERK9VecTokens" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE">
<span id="_CPPv3N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE"></span><span id="tensorrt_llm::executor::Request::kBatchedPostProcessorName__auto"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a5c087116d503c2f4a65f45db08b671da"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBatchedPostProcessorName</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;batched&quot;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request25kBatchedPostProcessorNameE" title="Link to this definition"></a><br /></dt>
<dd><p>This logits postprocessor name will dispatch to the batched logits postprocessor. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor7Request5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor7Request5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor7Request5mImplE"></span><span id="tensorrt_llm::executor::Request::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Request_1a2bef0e71b0eadc746b10462a419ff339"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor7Request5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6ResultE">
<span id="_CPPv3N12tensorrt_llm8executor6ResultE"></span><span id="_CPPv2N12tensorrt_llm8executor6ResultE"></span><span id="tensorrt_llm::executor::Result"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Result</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Struct that holds the generation result. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result7isFinalE">
<span id="_CPPv3N12tensorrt_llm8executor6Result7isFinalE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result7isFinalE"></span><span id="tensorrt_llm::executor::Result::isFinal__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1aa17894b95a897fc98e10059a95060097"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isFinal</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result7isFinalE" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if this is the final result for the request. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result14outputTokenIdsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result14outputTokenIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result14outputTokenIdsE"></span><span id="tensorrt_llm::executor::Result::outputTokenIds__BeamTokens"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a878f113efb1c2e087bece4205f61fbfb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="tensorrt_llm::executor::BeamTokens"><span class="n"><span class="pre">BeamTokens</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">outputTokenIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result14outputTokenIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The output tokens for each beam. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result11cumLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result11cumLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result11cumLogProbsE"></span><span id="tensorrt_llm::executor::Result::cumLogProbs__std::optional:VecLogProbs:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1aa4463aefb92ec56587efff7b64ca3f97"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE" title="tensorrt_llm::executor::VecLogProbs"><span class="n"><span class="pre">VecLogProbs</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cumLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result11cumLogProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>The cumulative log probabilities. Size beamSize. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result8logProbsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result8logProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result8logProbsE"></span><span id="tensorrt_llm::executor::Result::logProbs__std::optional:std::vector:VecLogProbs::"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a11e7bc33f082ca62a83dbc12a734889c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE" title="tensorrt_llm::executor::VecLogProbs"><span class="n"><span class="pre">VecLogProbs</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">logProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result8logProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>The log probabilities for each generated token. Size [beamSize, outputLen]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result13contextLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result13contextLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result13contextLogitsE"></span><span id="tensorrt_llm::executor::Result::contextLogits__std::optional:Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1aade7284f001e6d0418709492b3b90cf1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result13contextLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>The context logits. Size [promptLen, vocabSizePadded]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result16generationLogitsE">
<span id="_CPPv3N12tensorrt_llm8executor6Result16generationLogitsE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result16generationLogitsE"></span><span id="tensorrt_llm::executor::Result::generationLogits__std::optional:Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1a3c7678084cfb911c75b34ea2c4043d9b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">generationLogits</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result16generationLogitsE" title="Link to this definition"></a><br /></dt>
<dd><p>The context logits. Size [beamSize, maxNewTokens, vocabSizePadded]. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Result13encoderOutputE">
<span id="_CPPv3N12tensorrt_llm8executor6Result13encoderOutputE"></span><span id="_CPPv2N12tensorrt_llm8executor6Result13encoderOutputE"></span><span id="tensorrt_llm::executor::Result::encoderOutput__std::optional:Tensor:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1Result_1af1fe609c6a99ac08b1bf4c687ca8e654"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">encoderOutput</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Result13encoderOutputE" title="Link to this definition"></a><br /></dt>
<dd><p>The encoder output. Size [encoderLen, hiddenSize]. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseE">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseE"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseE"></span><span id="tensorrt_llm::executor::Response"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Class that holds either an error or a result. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringE">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringE"></span><span id="tensorrt_llm::executor::Response::Response__IdType.ss"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1ae173156b184a172bf102dfd3d5eab869"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestId</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">errorMsg</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdTypeNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6Result">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseE6IdType6Result"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseE6IdType6Result"></span><span id="tensorrt_llm::executor::Response::Response__IdType.Result"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1ac224d86a5ce1aa04d4c349941550ec40"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestId</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6Result" title="tensorrt_llm::executor::Response::Response::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">Result</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseE6IdType6Result" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseD0Ev"></span><span id="tensorrt_llm::executor::Response::~Response"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1aef44ea215e23b7f97c69b3de2e2d612b"></span><span class="sig-name descname"><span class="n"><span class="pre">~Response</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseERK8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseERK8Response"></span><span id="tensorrt_llm::executor::Response::Response__ResponseCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1aa59cf06a2cd20bcbb414c6329f950653"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response" title="tensorrt_llm::executor::Response::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERK8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response">
<span id="_CPPv3N12tensorrt_llm8executor8Response8ResponseERR8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8Response8ResponseERR8Response"></span><span id="tensorrt_llm::executor::Response::Response__ResponseRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a6775c21a72390415490ec5138a4c91c3"></span><span class="sig-name descname"><span class="n"><span class="pre">Response</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response" title="tensorrt_llm::executor::Response::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response8ResponseERR8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseaSERK8Response">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseaSERK8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseaSERK8Response"></span><span id="tensorrt_llm::executor::Response::assign-operator__ResponseCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a3bdf004f4b1dd9cf5ad39b34cfaa8063"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseaSERK8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ResponseaSERR8Response">
<span id="_CPPv3N12tensorrt_llm8executor8ResponseaSERR8Response"></span><span id="_CPPv2N12tensorrt_llm8executor8ResponseaSERR8Response"></span><span id="tensorrt_llm::executor::Response::assign-operator__ResponseRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a38b7418538a8499ed7a3329482ac1da0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ResponseaSERR8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response12getRequestIdEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response12getRequestIdEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response12getRequestIdEv"></span><span id="tensorrt_llm::executor::Response::getRequestIdC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1adf694176a77659e4f4afcb4fd7112eea"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRequestId</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response12getRequestIdEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the id of the request for which this response was generated. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response8hasErrorEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response8hasErrorEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response8hasErrorEv"></span><span id="tensorrt_llm::executor::Response::hasErrorC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1acd0e0fe2d71ecd4e9a6ae7529d5a9958"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">hasError</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response8hasErrorEv" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if this response has an error or not. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response11getErrorMsgEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response11getErrorMsgEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response11getErrorMsgEv"></span><span id="tensorrt_llm::executor::Response::getErrorMsgC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1ace8398f4c0adafbde1fb62320c10a138"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getErrorMsg</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response11getErrorMsgEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the error msg for this response Will throw an exception if hasError is false. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Response9getResultEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Response9getResultEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Response9getResultEv"></span><span id="tensorrt_llm::executor::Response::getResultC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1aeccfdc8c14c2b3ffb6b3ec61c9ab95be"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getResult</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Response9getResultEv" title="Link to this definition"></a><br /></dt>
<dd><p>Get the result for this response Will throw an exception if hasResult is true. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Response5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor8Response5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor8Response5mImplE"></span><span id="tensorrt_llm::executor::Response::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Response_1a62ec31b2113d81890e913e6b5e4fddac"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Response5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfigE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfigE"></span><span id="tensorrt_llm::executor::SchedulerConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SchedulerConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the scheduler. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEE"></span><span id="tensorrt_llm::executor::SchedulerConfig::SchedulerConfig__CapacitySchedulerPolicy.std::optional:ContextChunkingPolicy:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1a1ef556ebbeeaf195f3a9f6b8579852c9"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SchedulerConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">capacitySchedulerPolicy</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE" title="tensorrt_llm::executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT"><span class="n"><span class="pre">kGUARANTEED_NO_EVICT</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">contextChunkingPolicy</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig15SchedulerConfigE23CapacitySchedulerPolicyNSt8optionalI21ContextChunkingPolicyEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv">
<span id="_CPPv3NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv"></span><span id="tensorrt_llm::executor::SchedulerConfig::getCapacitySchedulerPolicyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1aa72f57d07d885b6fd256189911076b8b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCapacitySchedulerPolicy</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig26getCapacitySchedulerPolicyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv">
<span id="_CPPv3NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv"></span><span id="tensorrt_llm::executor::SchedulerConfig::getContextChunkingPolicyC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1ad7d4ddfc842da48456f972055beab467"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getContextChunkingPolicy</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15SchedulerConfig24getContextChunkingPolicyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE"></span><span id="tensorrt_llm::executor::SchedulerConfig::mCapacitySchedulerPolicy__CapacitySchedulerPolicy"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1a3c5dc1e328e369821e9b510bb89ddd0f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCapacitySchedulerPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig24mCapacitySchedulerPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p>The capacity scheduler policy. See CapacitySchedulerPolicy. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE"></span><span id="tensorrt_llm::executor::SchedulerConfig::mContextChunkingPolicy__std::optional:ContextChunkingPolicy:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1SchedulerConfig_1aa3f547ef1160624ae6e9ddce782967cf"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mContextChunkingPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfig22mContextChunkingPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p>The context chunking policy. See ContextChunkingPolicy. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfigE"></span><span id="tensorrt_llm::executor::KvCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the KV cache. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEb">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEb"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEb"></span><span id="tensorrt_llm::executor::KvCacheConfig::KvCacheConfig__b.std::optional:SizeType32:CR.std::optional:SizeType32:CR.std::optional:SizeType32:CR.std::optional:FloatType:CR.std::optional:s:CR.b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ad8ed75fe8bba135d3bb581e0e61718ad"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableBlockReuse</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">maxTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">maxAttentionWindow</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">sinkTokenLength</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">freeGpuMemoryFraction</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">hostCacheSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">onboardBlocks</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig13KvCacheConfigEbRKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI10SizeType32EERKNSt8optionalI9FloatTypeEERKNSt8optionalI6size_tEEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getEnableBlockReuseC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a215347e4303538281e8918d711bee0fe"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableBlockReuse</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig19getEnableBlockReuseEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getMaxTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a2bcb437b4335480c619edb23d397b2ac"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig12getMaxTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig21getMaxAttentionWindowEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig21getMaxAttentionWindowEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig21getMaxAttentionWindowEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getMaxAttentionWindowC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a6a63942c2ca28fa9be723255c39c1927"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxAttentionWindow</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig21getMaxAttentionWindowEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getSinkTokenLengthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1aea5f170576c1a9d8ee0049906cb93380"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSinkTokenLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig18getSinkTokenLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getFreeGpuMemoryFractionC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ae0c26c4b9a78e162620c1efac1ac8268"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getFreeGpuMemoryFraction</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig24getFreeGpuMemoryFractionEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getHostCacheSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a1756409572125a606876ce92a0820356"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getHostCacheSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getHostCacheSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv">
<span id="_CPPv3NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv"></span><span id="_CPPv2NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv"></span><span id="tensorrt_llm::executor::KvCacheConfig::getOnboardBlocksC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a596dc5ef42e0aaeffa728705972635f5"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOnboardBlocks</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor13KvCacheConfig16getOnboardBlocksEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb"></span><span id="tensorrt_llm::executor::KvCacheConfig::setEnableBlockReuse__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a934f7362c2d931489c7df5c86a5c19c1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEnableBlockReuse</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableBlockReuse</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig19setEnableBlockReuseEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32"></span><span id="tensorrt_llm::executor::KvCacheConfig::setMaxTokens__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1af2998d1bb93ae001bf468e8c6c78214c"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig12setMaxTokensE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig21setMaxAttentionWindowE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig21setMaxAttentionWindowE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig21setMaxAttentionWindowE10SizeType32"></span><span id="tensorrt_llm::executor::KvCacheConfig::setMaxAttentionWindow__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a9be00f0c2e90ae5fee0f7318cd0fb2ea"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxAttentionWindow</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxAttentionWindow</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig21setMaxAttentionWindowE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32"></span><span id="tensorrt_llm::executor::KvCacheConfig::setSinkTokenLength__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1abc33d9483367d58b2744884618840a37"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSinkTokenLength</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">sinkTokenLength</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig18setSinkTokenLengthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType"></span><span id="tensorrt_llm::executor::KvCacheConfig::setFreeGpuMemoryFraction__FloatType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1acf2e6b9f0c81b3c1a299e96a6737b587"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setFreeGpuMemoryFraction</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">freeGpuMemoryFraction</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig24setFreeGpuMemoryFractionE9FloatType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t"></span><span id="tensorrt_llm::executor::KvCacheConfig::setHostCacheSize__s"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ab23edc123b320ef59c4c4117e7c97117"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setHostCacheSize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">hostCacheSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setHostCacheSizeE6size_t" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb"></span><span id="tensorrt_llm::executor::KvCacheConfig::setOnboardBlocks__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ac1ff38e4da7ddf084bc4ad5192257193"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOnboardBlocks</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">onboardBlocks</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16setOnboardBlocksEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mEnableBlockReuse__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a60551af15587436a83fb1ca17a82878f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableBlockReuse</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig17mEnableBlockReuseE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if KV cache blocks can be reused for different requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mMaxTokens__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a06bf9092bf1b1abedc4c256edb1be59a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig10mMaxTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The maximum number of tokens that should be stored in the KV cache If both mMaxTokens and mFreeGpuMemoryFraction are specified, memory corresponding to the minimum will be allocated. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig19mMaxAttentionWindowE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig19mMaxAttentionWindowE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig19mMaxAttentionWindowE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mMaxAttentionWindow__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1aa147d131b0ac1c42f0282ae76b6077f2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxAttentionWindow</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig19mMaxAttentionWindowE" title="Link to this definition"></a><br /></dt>
<dd><p>Size of the attention window for each sequence. Only the last mMaxAttentionWindow tokens of each sequence will be stored in the KV cache. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mSinkTokenLength__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a84f967e8a537c1060870714612acc0e4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSinkTokenLength</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig16mSinkTokenLengthE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of sink tokens (tokens to always keep in attention window) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mFreeGpuMemoryFraction__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ac48a05dd2ed2669970f12495e49c3cd1"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mFreeGpuMemoryFraction</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig22mFreeGpuMemoryFractionE" title="Link to this definition"></a><br /></dt>
<dd><p>The fraction of GPU memory fraction that should be allocated for the KV cache. Default is 90%. If both mMaxTokens and mFreeGpuMemoryFraction are specified, memory corresponding to the minimum will be allocated. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mHostCacheSize__std::optional:s:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1a3ec456502080d1b581b3619ce46a0ba6"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mHostCacheSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mHostCacheSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>Size of secondary memory pool in bytes. Default is 0. Having a secondary memory pool increases KV cache block reuse potential. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE"></span><span id="tensorrt_llm::executor::KvCacheConfig::mOnboardBlocks__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1KvCacheConfig_1ad16b0f0939192c1b97e6d2e905c3a1cc"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOnboardBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfig14mOnboardBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls whether offloaded blocks should be onboarded back into primary memory before being reused. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfigE"></span><span id="tensorrt_llm::executor::OrchestratorConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OrchestratorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::OrchestratorConfig__b.ss.std::shared_ptr:mpi::MpiComm:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a1a41bee27ec8841079002ce994e11c43"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">OrchestratorConfig</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">isOrchestrator</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">workerExecutablePath</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="s"><span class="pre">&quot;&quot;</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">orchLeaderComm</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig18OrchestratorConfigEbNSt6stringENSt10shared_ptrIN3mpi7MpiCommEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getIsOrchestratorC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a0faa15a0f3ce88cb25eb07a1e1a91d34"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getIsOrchestrator</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getIsOrchestratorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getWorkerExecutablePathC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a603abfa699a11b7e910278cc9c027542"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWorkerExecutablePath</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig23getWorkerExecutablePathEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv">
<span id="_CPPv3NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv"></span><span id="_CPPv2NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv"></span><span id="tensorrt_llm::executor::OrchestratorConfig::getOrchLeaderCommC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a826ad0ee9132a97156e928eb4c8dce32"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOrchLeaderComm</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor18OrchestratorConfig17getOrchLeaderCommEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setIsOrchestrator__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a5384e155759356518a2bca388a1ae1d0"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setIsOrchestrator</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">isOrchestrator</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setIsOrchestratorEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setWorkerExecutablePath__ssCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a557913170113b0f37daab422bb8552d1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setWorkerExecutablePath</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">workerExecutablePath</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig23setWorkerExecutablePathERKNSt6stringE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::setOrchLeaderComm__std::shared_ptr:mpi::MpiComm:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1ab4e88541c7a38d3675485f227030f7e2"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOrchLeaderComm</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchLeaderComm</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig17setOrchLeaderCommERKNSt10shared_ptrIN3mpi7MpiCommEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mIsOrchestrator__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a272f1d1692fae68ed8242d8180326977"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mIsOrchestrator</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mIsOrchestratorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mWorkerExecutablePath__ss"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a8a4c20267c20632d8107e9b8041198c2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWorkerExecutablePath</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig21mWorkerExecutablePathE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE">
<span id="_CPPv3N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE"></span><span id="_CPPv2N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE"></span><span id="tensorrt_llm::executor::OrchestratorConfig::mOrchLeaderComm__std::shared_ptr:mpi::MpiComm:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1OrchestratorConfig_1a16ac5a458ad5e4e048d0badcba4b54d5"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm3mpiE" title="tensorrt_llm::mpi"><span class="n"><span class="pre">mpi</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">MpiComm</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOrchLeaderComm</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfig15mOrchLeaderCommE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfigE"></span><span id="tensorrt_llm::executor::ParallelConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ParallelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>A configuration class for the parallel execution parameters Currently only supports commType = CommunicationType::kMPI. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE"></span><span id="tensorrt_llm::executor::ParallelConfig::ParallelConfig__CommunicationType.CommunicationMode.std::optional:std::vector:SizeType32::.std::optional:std::vector:SizeType32::.std::optional:OrchestratorConfig:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a3f4a74b658b6e839fe93f65a7e03a382"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ParallelConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">commType</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE" title="tensorrt_llm::executor::CommunicationType::kMPI"><span class="n"><span class="pre">kMPI</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">commMode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE" title="tensorrt_llm::executor::CommunicationMode::kLEADER"><span class="n"><span class="pre">kLEADER</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">deviceIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">participantIds</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig14ParallelConfigE17CommunicationType17CommunicationModeNSt8optionalINSt6vectorI10SizeType32EEEENSt8optionalINSt6vectorI10SizeType32EEEERKNSt8optionalI18OrchestratorConfigEE" title="Link to this definition"></a><br /></dt>
<dd><p>Constructor. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>commType</strong> – The communication type. See CommunicationType. </p></li>
<li><p><strong>commMode</strong> – The communication mode. See CommunicationMode. </p></li>
<li><p><strong>deviceIds</strong> – The IDs of the GPUs involved in the execution of the model </p></li>
<li><p><strong>participantIds</strong> – The participant IDs (MPI ranks if commType == kMPI) involved in the execution of the model. The first participant is considered to be the leader. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getCommunicationTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ad384d17a72f351d994e604594d6a6f4b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCommunicationType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getCommunicationModeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a9bf12e33fc86542ee77ff1488fa7835c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getCommunicationMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig20getCommunicationModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getDeviceIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ac61266beceba80f61e4e4ccbaecc15ba"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDeviceIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig12getDeviceIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getParticipantIdsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a69a2e829c07bcb9f087999b20adb6a2c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParticipantIds</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig17getParticipantIdsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv"></span><span id="tensorrt_llm::executor::ParallelConfig::getOrchestratorConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1aeeddcfa32b3a9b243bb49a6f6f53c153"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOrchestratorConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ParallelConfig21getOrchestratorConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType"></span><span id="tensorrt_llm::executor::ParallelConfig::setCommunicationType__CommunicationType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ad16a96f04fa3474bf67452910f64a8bc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCommunicationType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">type</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationTypeE17CommunicationType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode"></span><span id="tensorrt_llm::executor::ParallelConfig::setCommunicationMode__CommunicationMode"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a1bb2819d5c1add8908d9db259b16c741"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setCommunicationMode</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">mode</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig20setCommunicationModeE17CommunicationMode" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE"></span><span id="tensorrt_llm::executor::ParallelConfig::setDeviceIds__std::vector:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a6bcd6e1acdde379946d599e884fb7ee1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDeviceIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">deviceIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig12setDeviceIdsERKNSt6vectorI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE"></span><span id="tensorrt_llm::executor::ParallelConfig::setParticipantIds__std::vector:SizeType32:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a7e3f6c11ef029cdee705305ca1129a0a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setParticipantIds</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">participantIds</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig17setParticipantIdsERKNSt6vectorI10SizeType32EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig"></span><span id="tensorrt_llm::executor::ParallelConfig::setOrchestratorConfig__OrchestratorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a72057d9d07087fc7b6fdb2795bf1fb5e"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setOrchestratorConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig21setOrchestratorConfigERK18OrchestratorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommTypeE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig9mCommTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig9mCommTypeE"></span><span id="tensorrt_llm::executor::ParallelConfig::mCommType__CommunicationType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ae1b162dc41f8541aba52d90c6bd76d18"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="tensorrt_llm::executor::CommunicationType"><span class="n"><span class="pre">CommunicationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The type of communication protocol used. Default is MPI. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommModeE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig9mCommModeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig9mCommModeE"></span><span id="tensorrt_llm::executor::ParallelConfig::mCommMode__CommunicationMode"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ad5b06cc50716576658e25aa7d30fd6b6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="tensorrt_llm::executor::CommunicationMode"><span class="n"><span class="pre">CommunicationMode</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mCommMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig9mCommModeE" title="Link to this definition"></a><br /></dt>
<dd><p>The mode of communication. See CommunicationMode. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE"></span><span id="tensorrt_llm::executor::ParallelConfig::mDeviceIds__std::optional:std::vector:SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a4a9207031d67fbe267d38d89c8901e81"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDeviceIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig10mDeviceIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The GPU device ids to use for executing this model. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE"></span><span id="tensorrt_llm::executor::ParallelConfig::mParticipantIds__std::optional:std::vector:SizeType32::"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1ae0e3d3936b187b3fed26b16220ee8a57"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mParticipantIds</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig15mParticipantIdsE" title="Link to this definition"></a><br /></dt>
<dd><p>The participant ids (MPI ranks for example) used for executing this model. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE"></span><span id="tensorrt_llm::executor::ParallelConfig::mOrchestratorConfig__std::optional:OrchestratorConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ParallelConfig_1a0a8ccedf7de365ca80be267e90627954"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOrchestratorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfig19mOrchestratorConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Optional orchestrator configuration. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfigE"></span><span id="tensorrt_llm::executor::PeftCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PeftCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>config for PeftCacheManager </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::PeftCacheConfig__SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.SizeType32.std::optional:float:CR.std::optional:s:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a5cd90c5f813eb4b72b8000bd6459a1c8"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">PeftCacheConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numHostModuleLayer</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numDeviceModuleLayer</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">optimalAdapterSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxAdapterSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">64</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numPutWorkers</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numEnsureWorkers</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">numCopyStreams</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxPagesPerBlockHost</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">24</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxPagesPerBlockDevice</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">deviceCachePercent</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">hostCacheSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15PeftCacheConfigE10SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType3210SizeType32RKNSt8optionalIfEERKNSt8optionalI6size_tEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig"></span><span id="tensorrt_llm::executor::PeftCacheConfig::eq-operator__PeftCacheConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ab60292786cb2f414b52a8d571708a281"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfigeqERK15PeftCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumHostModuleLayerC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ad0a03ff1b983f8c306912156a370d436"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumHostModuleLayer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getNumHostModuleLayerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumDeviceModuleLayerC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ac6585fa80da031f50cfd241dab5516d4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumDeviceModuleLayer</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getNumDeviceModuleLayerEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getOptimalAdapterSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a550e64682974fb60078a5c858d84bcc2"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getOptimalAdapterSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getOptimalAdapterSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getMaxAdapterSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a7e11e64fc78e41f098df0a9b2d683939"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxAdapterSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getMaxAdapterSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumPutWorkersC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1af92e38b38cb61de1dad2210a76011163"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumPutWorkers</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getNumPutWorkersEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumEnsureWorkersC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abcaf1eefd4d9ef5fbffa47eb885b1b6d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumEnsureWorkers</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig19getNumEnsureWorkersEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getNumCopyStreamsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a3e28ede570906ef9c3ccfc2898e0e4aa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumCopyStreams</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig17getNumCopyStreamsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getMaxPagesPerBlockHostC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1af7ab8d350fee59f9890e048393bf5a02"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxPagesPerBlockHost</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig23getMaxPagesPerBlockHostEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getMaxPagesPerBlockDeviceC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1aa7147fa1d4bbef5ce8cc038b5434cc8e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxPagesPerBlockDevice</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig25getMaxPagesPerBlockDeviceEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getDeviceCachePercentC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a73e55c7617fe5b5645738965050ecad8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDeviceCachePercent</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig21getDeviceCachePercentEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv"></span><span id="tensorrt_llm::executor::PeftCacheConfig::getHostCacheSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ac58edd4bc1b6d64e88eb01810d43e680"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getHostCacheSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor15PeftCacheConfig16getHostCacheSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumHostModuleLayer__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ae3a957e6c10ff32fda9cdfa261307a23"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumHostModuleLayer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mNumHostModuleLayerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumDeviceModuleLayer__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a6c734def4b320c40ec7758a2f9f2f928"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumDeviceModuleLayer</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mNumDeviceModuleLayerE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mOptimalAdapterSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a4cbcea2497e456f7f5c6988d9afba569"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mOptimalAdapterSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mOptimalAdapterSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mMaxAdapterSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abe38d2b75a813f5b2e73e5800083984a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxAdapterSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mMaxAdapterSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumPutWorkers__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1ab5b4bc29cc88c7cc3dd7f90bd8632966"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumPutWorkers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mNumPutWorkersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumEnsureWorkers__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a0e10d459eaf84adcd6d50926775f5a45"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumEnsureWorkers</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig17mNumEnsureWorkersE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mNumCopyStreams__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a2d7ccbe5d9c11d02cab32a1a7557c511"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNumCopyStreams</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig15mNumCopyStreamsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mMaxPagesPerBlockHost__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a968719a3d4bec709899c4bbb50e84a6f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxPagesPerBlockHost</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig21mMaxPagesPerBlockHostE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mMaxPagesPerBlockDevice__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1abd1029f5855b119ea0b6c66c46e59eee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxPagesPerBlockDevice</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig23mMaxPagesPerBlockDeviceE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mDeviceCachePercent__std::optional:FloatType:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a838668e3174dd46dddb8bb60a9da0e05"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDeviceCachePercent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig19mDeviceCachePercentE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE">
<span id="_CPPv3N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE"></span><span id="tensorrt_llm::executor::PeftCacheConfig::mHostCacheSize__std::optional:s:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1PeftCacheConfig_1a656930138cb6f162e7fc15c82778d1cc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">size_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mHostCacheSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfig14mHostCacheSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfigE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::LookaheadDecodingConfig__SizeType32.SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1abef9e49b2e2e6312cf4c280146d24c0a"></span><span class="sig-name descname"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">windowSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ngramSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">verificationSetSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigE10SizeType3210SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::LookaheadDecodingConfig"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a2affd5248374a476986c6aadc51eac4f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig23LookaheadDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::eq-operator__LookaheadDecodingConfigCRC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a419b3ac66d9136151a2a0b7d526ad9b3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfigeqERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1af70d4453310e9f2e1bcdad7dd8bfc935"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">tuple</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">get</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig3getEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getWindowSizeC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a8aa7577b18ff38bf4bc9640acf9aae47"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getWindowSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig13getWindowSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getNgramSizeC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a5b20a75ca2a5e7f08259f0500330e58e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNgramSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig12getNgramSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::getVerificationSetSizeC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1adad2572691ad3764b01c48b8ba2885c0"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getVerificationSetSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig22getVerificationSetSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::calculateSpeculativeResourceC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a51f7fd2a65379380180ec15a2908ab45"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">tuple</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">calculateSpeculativeResource</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig28calculateSpeculativeResourceEv" title="Link to this definition"></a><br /></dt>
<dd><p>return &lt;maxDecodingTokens, maxPathLen, maxDraftTokens, maxDraftPathLen&gt; </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::isLE__LookaheadDecodingConfigCRC"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a87a081925bb740bf34f8c3a0a65f4180"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLE</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">that</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor23LookaheadDecodingConfig4isLEERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>return true when <code class="docutils literal notranslate"><span class="pre">this</span></code> can be executed on resources defined by <code class="docutils literal notranslate"><span class="pre">that</span></code></p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::isLegal__SizeType32.SizeType32.SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1aec6d6b6c329339fe92c04b3c99606c27"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLegal</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">windowSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">ngramSize</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">verificationSetSize</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig7isLegalE10SizeType3210SizeType3210SizeType32" title="Link to this definition"></a><br /></dt>
<dd><p>return true when the parameter combination is valid. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::mWindowSize__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a5325178fb1f33a47552ac21f4315e6fa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mWindowSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig11mWindowSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::mNgramSize__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a47e578a6e17991ce817a1763cf610e57"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNgramSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig10mNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE">
<span id="_CPPv3N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE"></span><span id="tensorrt_llm::executor::LookaheadDecodingConfig::mVerificationSetSize__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1LookaheadDecodingConfig_1a01deb81773d9c4ccf37074d773f5ca34"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mVerificationSetSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfig20mVerificationSetSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfigE"></span><span id="tensorrt_llm::executor::DecodingConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the speculative decoding. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEE"></span><span id="tensorrt_llm::executor::DecodingConfig::DecodingConfig__std::optional:DecodingMode:.std::optional:LookaheadDecodingConfig:.std::optional:MedusaChoices:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a394692357f1f116a76d0725b589641f8"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decodingMode</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">medusaChoices</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14DecodingConfigENSt8optionalI12DecodingModeEENSt8optionalI23LookaheadDecodingConfigEENSt8optionalI13MedusaChoicesEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig"></span><span id="tensorrt_llm::executor::DecodingConfig::eq-operator__DecodingConfigCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a72aae341b22cfafe152513cf5570c39f"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfigeqERK14DecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode"></span><span id="tensorrt_llm::executor::DecodingConfig::setDecodingMode__DecodingModeCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a67e4a10d6f2bc039bd1a22d86a54b13b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDecodingMode</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig15setDecodingModeERK12DecodingMode" title="Link to this definition"></a><br /></dt>
<dd><p>Sets decoding mode. Some modes require the use of their own setters. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getDecodingModeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a01ab2e2e8f168e44fbc5f3260c4417be"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDecodingMode</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig15getDecodingModeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::DecodingConfig::setLookaheadDecoding__LookaheadDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a4e9181c201fcd72009a337bb706efc4f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLookaheadDecoding</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig20setLookaheadDecodingERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd><p>Sets lookahead decoding mode and config. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getLookaheadDecodingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1ab50f36a4e3ec50b68f49b98ba6fe4092"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig26getLookaheadDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices"></span><span id="tensorrt_llm::executor::DecodingConfig::setMedusaChoices__MedusaChoicesCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1aa970dcb3b1ce45250e4f33febd6f20e8"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMedusaChoices</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig16setMedusaChoicesERK13MedusaChoices" title="Link to this definition"></a><br /></dt>
<dd><p>Sets medusa mode and config. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv">
<span id="_CPPv3NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv"></span><span id="tensorrt_llm::executor::DecodingConfig::getMedusaChoicesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1ab302eed5529b44f9f798f6b47502c1a4"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMedusaChoices</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14DecodingConfig16getMedusaChoicesEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE"></span><span id="tensorrt_llm::executor::DecodingConfig::mDecodingMode__std::optional:DecodingMode:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a44b33dcb361289657af45112b7d7ef65"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig13mDecodingModeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE"></span><span id="tensorrt_llm::executor::DecodingConfig::mLookaheadDecodingConfig__std::optional:LookaheadDecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a714a527e462727b8aba4c8ff4e18c35e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLookaheadDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig24mLookaheadDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE">
<span id="_CPPv3N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE"></span><span id="_CPPv2N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE"></span><span id="tensorrt_llm::executor::DecodingConfig::mMedusaChoices__std::optional:MedusaChoices:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingConfig_1a13bb97024510b32cbb56b1606742e7ec"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="tensorrt_llm::executor::MedusaChoices"><span class="n"><span class="pre">MedusaChoices</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMedusaChoices</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfig14mMedusaChoicesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExecutorConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Configuration class for the model executor. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType32RK15SchedulerConfigRK13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEENSt8optionalI14DecodingConfigEEf">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType32RK15SchedulerConfigRK13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEENSt8optionalI14DecodingConfigEEf"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType32RK15SchedulerConfigRK13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEENSt8optionalI14DecodingConfigEEf"></span><span id="tensorrt_llm::executor::ExecutorConfig::ExecutorConfig__SizeType32.SchedulerConfigCR.KvCacheConfigCR.b.b.SizeType32.SizeType32.BatchingType.std::optional:SizeType32:.std::optional:SizeType32:.std::optional:ParallelConfig:.std::optional:PeftCacheConfig:CR.std::optional:LogitsPostProcessorMap:.std::optional:LogitsPostProcessorBatched:.std::optional:DecodingConfig:.float"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a86d14bda433237ded02dcb06f794c372"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExecutorConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="p"><span class="pre">(</span></span><span class="p"><span class="pre">)</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableChunkedContext</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">normalizeLogProbs</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">true</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">iterStatsMaxIterations</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor30kDefaultIterStatsMaxIterationsE" title="tensorrt_llm::executor::kDefaultIterStatsMaxIterations"><span class="n"><span class="pre">kDefaultIterStatsMaxIterations</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestStatsMaxIterations</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor33kDefaultRequestStatsMaxIterationsE" title="tensorrt_llm::executor::kDefaultRequestStatsMaxIterations"><span class="n"><span class="pre">kDefaultRequestStatsMaxIterations</span></span></a>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchingType</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE" title="tensorrt_llm::executor::BatchingType::kINFLIGHT"><span class="n"><span class="pre">kINFLIGHT</span></span></a>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">parallelConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsPostProcessorMap</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">logitsPostProcessorBatched</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decodingConfig</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span>, <span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">gpuWeightsPercent</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14ExecutorConfigE10SizeType32RK15SchedulerConfigRK13KvCacheConfigbb10SizeType3210SizeType3212BatchingTypeNSt8optionalI10SizeType32EENSt8optionalI10SizeType32EENSt8optionalI14ParallelConfigEERKNSt8optionalI15PeftCacheConfigEENSt8optionalI22LogitsPostProcessorMapEENSt8optionalI26LogitsPostProcessorBatchedEENSt8optionalI14DecodingConfigEEf" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxBeamWidthC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af28528ea0a4f5d8252d5cf4aa809cab9"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxBeamWidth</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBeamWidthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getSchedulerConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ad04adf7773f5dc011df7811ed71a2065"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSchedulerConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getSchedulerConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getKvCacheConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a81554cd8e28a7c975dcb88e0bc678829"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getKvCacheConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig16getKvCacheConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getEnableChunkedContextC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acebbdef0320ea601a14d69ee84b0d8e8"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getEnableChunkedContext</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig23getEnableChunkedContextEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getNormalizeLogProbsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af9638fec77c9ad7a4a9230c6b8ccd3e2"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNormalizeLogProbs</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getNormalizeLogProbsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getIterStatsMaxIterationsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ad68d1ec95040a53535b2df4bbc5e67ad"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getIterStatsMaxIterations</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getIterStatsMaxIterationsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getRequestStatsMaxIterationsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a80fbe7fc988029b4e5605bc79d8983e5"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRequestStatsMaxIterations</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig28getRequestStatsMaxIterationsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getBatchingTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac90fe84eae462d1eae8d2ad84c4c8bee"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getBatchingType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getBatchingTypeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxBatchSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a229414863d46d182d8fbfea8ed16ed86"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxBatchSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxBatchSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getMaxNumTokensC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1adf2cfcd2066a5616cd0fcd27bb5fe015"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMaxNumTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig15getMaxNumTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getParallelConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af2a0d17c04fa958e9c82e1a2bbdb943d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getParallelConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getParallelConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getPeftCacheConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a1b2c3c4beec6025e380f211321e65a7b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getPeftCacheConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig18getPeftCacheConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getLogitsPostProcessorMapEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig25getLogitsPostProcessorMapEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig25getLogitsPostProcessorMapEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getLogitsPostProcessorMapC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a8173d1bdbb53547f1ee0b9be0ff54d3a"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogitsPostProcessorMap</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig25getLogitsPostProcessorMapEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig29getLogitsPostProcessorBatchedEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig29getLogitsPostProcessorBatchedEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig29getLogitsPostProcessorBatchedEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getLogitsPostProcessorBatchedC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a8fa78650d1d3d53a4e286d0d1734bffd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLogitsPostProcessorBatched</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig29getLogitsPostProcessorBatchedEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getDecodingConfigC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a581a6103b225b7857de92a61584969e8"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig17getDecodingConfigEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv">
<span id="_CPPv3NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv"></span><span id="_CPPv2NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv"></span><span id="tensorrt_llm::executor::ExecutorConfig::getGpuWeightsPercentC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a45340052913aba3d6e3acdf12b07d4fd"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getGpuWeightsPercent</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor14ExecutorConfig20getGpuWeightsPercentEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a4a9426ca428d09d9514e85a0c2b612f5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxBeamWidth</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBeamWidth</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBeamWidthE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxBatchSize__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acbccae1669e096e7214a532e8ca415dc"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxBatchSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxBatchSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxBatchSizeE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setMaxNumTokens__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af273f0c3f408e7b54354b00a9d69ca3a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setMaxNumTokens</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">maxNumTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setMaxNumTokensE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setSchedulerConfig__SchedulerConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a3d0b81d8422cbb579ba37d425c600004"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setSchedulerConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setSchedulerConfigERK15SchedulerConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setKvCacheConfig__KvCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1abdfb27a2cc58da0a28ce5c2486ea688f"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setKvCacheConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16setKvCacheConfigERK13KvCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb"></span><span id="tensorrt_llm::executor::ExecutorConfig::setEnableChunkedContext__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a6426dbe445f65be08fae30ddc03135f4"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setEnableChunkedContext</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">enableChunkedContext</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23setEnableChunkedContextEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb"></span><span id="tensorrt_llm::executor::ExecutorConfig::setNormalizeLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aa40eff9c23c0abd6d4be7977bb80d80a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setNormalizeLogProbs</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">normalizeLogProbs</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setNormalizeLogProbsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setIterStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acab1aa5f729a9d145d23c0a0c8c2e526"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setIterStatsMaxIterations</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">iterStatsMaxIterations</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setIterStatsMaxIterationsE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32"></span><span id="tensorrt_llm::executor::ExecutorConfig::setRequestStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1acbb1bfe2c8a77c80d3ec6e55ff00196a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setRequestStatsMaxIterations</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestStatsMaxIterations</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig28setRequestStatsMaxIterationsE10SizeType32" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType"></span><span id="tensorrt_llm::executor::ExecutorConfig::setBatchingType__BatchingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7171a21854120d745c185d6d2ecd9673"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBatchingType</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">batchingType</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15setBatchingTypeE12BatchingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setParallelConfig__ParallelConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a36697066f6b8a8e7ea8d031ce1bfdaca"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setParallelConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">parallelConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setParallelConfigERK14ParallelConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setPeftCacheConfig__PeftCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1af5453098e1af9097cc47acb28d1df39a"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setPeftCacheConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18setPeftCacheConfigERK15PeftCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setLogitsPostProcessorMapERK22LogitsPostProcessorMap">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig25setLogitsPostProcessorMapERK22LogitsPostProcessorMap"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig25setLogitsPostProcessorMapERK22LogitsPostProcessorMap"></span><span id="tensorrt_llm::executor::ExecutorConfig::setLogitsPostProcessorMap__LogitsPostProcessorMapCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aa1daf96b4f9492034e301623afe71b92"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLogitsPostProcessorMap</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logitsPostProcessorMap</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig25setLogitsPostProcessorMapERK22LogitsPostProcessorMap" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig29setLogitsPostProcessorBatchedERK26LogitsPostProcessorBatched">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig29setLogitsPostProcessorBatchedERK26LogitsPostProcessorBatched"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig29setLogitsPostProcessorBatchedERK26LogitsPostProcessorBatched"></span><span id="tensorrt_llm::executor::ExecutorConfig::setLogitsPostProcessorBatched__LogitsPostProcessorBatchedCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac6eca48ffd534911619b343c9d861682"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setLogitsPostProcessorBatched</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">logitsPostProcessorBatched</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig29setLogitsPostProcessorBatchedERK26LogitsPostProcessorBatched" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig"></span><span id="tensorrt_llm::executor::ExecutorConfig::setDecodingConfig__DecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a7d6173ec1a87d222ff1674d047b3d69b"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setDecodingConfig</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig17setDecodingConfigERK14DecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf"></span><span id="tensorrt_llm::executor::ExecutorConfig::setGpuWeightsPercent__floatCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a555f875ac320973a0e0bc94e0e9611b1"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setGpuWeightsPercent</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">gpuWeightsPercent</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig20setGpuWeightsPercentERKf" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxBeamWidth__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a94c48a5d5ec190006f387c7a2b6704a8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxBeamWidth</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBeamWidthE" title="Link to this definition"></a><br /></dt>
<dd><p>The beam width value of requests that will be sent to the executor. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mSchedulerConfig__SchedulerConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ab1625bee6fca601abb5f1eb6b6355b5b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mSchedulerConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mSchedulerConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The scheduler configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mKvCacheConfig__KvCacheConfig"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a9c032445833352f9d7878d16a145ed4f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mKvCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig14mKvCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The KV cache configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mEnableChunkedContext__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a82112a8e221adc965968a1f127ba473e"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mEnableChunkedContext</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig21mEnableChunkedContextE" title="Link to this definition"></a><br /></dt>
<dd><p>The KV cache configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mNormalizeLogProbs__b"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a84ac3e0d9136d145a47d24f5eef306c3"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mNormalizeLogProbs</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mNormalizeLogProbsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls if log probabilities should be normalized or not. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mIterStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a201b184dee2e18455e22f45e645ac113"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mIterStatsMaxIterations</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mIterStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the maximum number of iterations for which to keep statistics. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mRequestStatsMaxIterations__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a185a12d03febdc22bf56aedb1b05a80b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mRequestStatsMaxIterations</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig26mRequestStatsMaxIterationsE" title="Link to this definition"></a><br /></dt>
<dd><p>Controls the maximum number of iterations for which to keep per-request statistics. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mBatchingType__BatchingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a59f638d7780b319b27c606352db470d3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="tensorrt_llm::executor::BatchingType"><span class="n"><span class="pre">BatchingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mBatchingType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mBatchingTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The type of batching strategy to use. See BatchingType. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxBatchSize__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a02960bc64feef914f78fc0712796535f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxBatchSize</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxBatchSizeE" title="Link to this definition"></a><br /></dt>
<dd><p>The max batch size of requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mMaxNumTokens__std::optional:SizeType32:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a0b08ae45ad7fadece12e0112f0a6e06b"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mMaxNumTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig13mMaxNumTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The max number of tokens per batch. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mParallelConfig__std::optional:ParallelConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a723466ec6c98eb16b4e8afc69092702d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mParallelConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mParallelConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>The parallel execution configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mPeftCacheConfig__std::optional:PeftCacheConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1aa36980dc3db09507bf58db6d5323294d"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mPeftCacheConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig16mPeftCacheConfigE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mLogitsPostProcessorMapE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig23mLogitsPostProcessorMapE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig23mLogitsPostProcessorMapE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mLogitsPostProcessorMap__std::optional:LogitsPostProcessorMap:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a872f1012f5af2acc88851ae2714c5aed"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="tensorrt_llm::executor::LogitsPostProcessorMap"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogitsPostProcessorMap</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig23mLogitsPostProcessorMapE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig27mLogitsPostProcessorBatchedE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig27mLogitsPostProcessorBatchedE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig27mLogitsPostProcessorBatchedE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mLogitsPostProcessorBatched__std::optional:LogitsPostProcessorBatched:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1ac511291a66d9f2b4458a558018153d5c"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="tensorrt_llm::executor::LogitsPostProcessorBatched"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mLogitsPostProcessorBatched</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig27mLogitsPostProcessorBatchedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mDecodingConfig__std::optional:DecodingConfig:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a0861d3ae711ae600f5f6596128990a66"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mDecodingConfig</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig15mDecodingConfigE" title="Link to this definition"></a><br /></dt>
<dd><p>Decoding configuration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE">
<span id="_CPPv3N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE"></span><span id="_CPPv2N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE"></span><span id="tensorrt_llm::executor::ExecutorConfig::mGpuWeightsPercent__float"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1ExecutorConfig_1a418f075a4e8f30f52c45cbb6cffec2d5"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mGpuWeightsPercent</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfig18mGpuWeightsPercentE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ExecutorE">
<span id="_CPPv3N12tensorrt_llm8executor8ExecutorE"></span><span id="_CPPv2N12tensorrt_llm8executor8ExecutorE"></span><span id="tensorrt_llm::executor::Executor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ExecutorE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>The executor is responsible for receiving new requests and sending responses, and running the inference. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::filesystem::pathCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a74e112aaa1be8f0b62f8d96f59021e1a"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">modelPath</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd><dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>modelPath</strong> – Path to the folder that defines the model to run </p></li>
<li><p><strong>modelType</strong> – The type of model </p></li>
<li><p><strong>executorConfig</strong> – The configuration for the executor </p></li>
<li><p><strong>comm</strong> – An optional inter-process communicator configuration </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::filesystem::pathCR.std::filesystem::pathCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af2fb4913e74a0cc0b3a85ef58bd1f662"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderModelPath</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">filesystem</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">path</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decoderModelPath</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt10filesystem4pathERKNSt10filesystem4pathE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::vector:uint8_t:CR.ssCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a77cf7b361572e94cd5eb17e06bd92450"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">engineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">jsonConfigStr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::vector:uint8_t:CR.ssCR.std::vector:uint8_t:CR.ssCR.ModelType.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a4186a331191390870fb8b8c98d4f78da"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderEngineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">encoderJsonConfigStr</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decoderEngineBuffer</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decoderJsonConfigStr</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">modelType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorERKNSt6vectorI7uint8_tEERKNSt6stringERKNSt6vectorI7uint8_tEERKNSt6stringE9ModelTypeRK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::shared_ptr:Model:.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a9a98e9c79f95e75c2a16f900986082ce"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Model</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">model</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEERK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Executor::Executor__std::shared_ptr:Model:.std::shared_ptr:Model:.ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1aedb3a745955f57fa268a326e49c0f741"></span><span class="sig-name descname"><span class="n"><span class="pre">Executor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Model</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">encoderModel</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Model</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">decoderModel</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8ExecutorENSt10shared_ptrI5ModelEENSt10shared_ptrI5ModelEERK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8ExecutorD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor8ExecutorD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor8ExecutorD0Ev"></span><span id="tensorrt_llm::executor::Executor::~Executor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a7c5a64a1cc125f8b6a17f05c85530b57"></span><span class="sig-name descname"><span class="n"><span class="pre">~Executor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8ExecutorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request"></span><span id="tensorrt_llm::executor::Executor::enqueueRequest__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1af760463b7371a09773207283f9f0fbb6"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">enqueueRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14enqueueRequestERK7Request" title="Link to this definition"></a><br /></dt>
<dd><p>Enqueue a new request. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>request</strong> – The LLM request which contains input tokens and request parameters </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A unique id that identifies the request </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE"></span><span id="tensorrt_llm::executor::Executor::enqueueRequests__std::vector:Request:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ab0914245aaccbe66bcfa76fd025666bc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">enqueueRequests</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requests</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor15enqueueRequestsERKNSt6vectorI7RequestEE" title="Link to this definition"></a><br /></dt>
<dd><p>Enqueue a batch of request. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::Executor::awaitResponses__std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1abf1008b7d8f7dba2f78e13636c565dc5"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for ready responses. </p>
<p><div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">This</span> <span class="n">overload</span> <span class="n">awaits</span> <span class="k">for</span> <span class="nb">any</span> <span class="n">ready</span> <span class="n">responses</span><span class="o">.</span> <span class="n">In</span> <span class="n">particular</span><span class="p">,</span> <span class="k">if</span> <span class="n">several</span> <span class="n">requests</span>
<span class="n">have</span> <span class="n">been</span> <span class="n">enqueued</span><span class="p">,</span> <span class="n">this</span> <span class="n">method</span> <span class="n">will</span> <span class="n">provide</span> <span class="nb">any</span> <span class="n">ready</span> <span class="n">responses</span> <span class="n">without</span> <span class="n">order</span> <span class="n">guarantees</span><span class="o">.</span>
</pre></div>
</div>
</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>timeout</strong> – The maximum time to wait for new responses </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::Executor::awaitResponses__IdTypeCR.std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a0c2e8f2117ec7d3de830c04b603226dc"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitResponses</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestId</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERK6IdTypeRKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for ready responses. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>id</strong> – A request id </p></li>
<li><p><strong>timeout</strong> – The maximum time to wait for new responses </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE"></span><span id="tensorrt_llm::executor::Executor::awaitResponses__std::vector:IdType:CR.std::optional:std::chrono::milliseconds:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a171de1a090dc673cac57a832dde58738"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">awaitResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestIds</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">chrono</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">milliseconds</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">timeout</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor14awaitResponsesERKNSt6vectorI6IdTypeEERKNSt8optionalINSt6chrono12millisecondsEEE" title="Link to this definition"></a><br /></dt>
<dd><p>Await for multiple ready responses. </p>
<p><div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="n">A</span> <span class="n">multiple</span> <span class="n">ID</span> <span class="n">request</span> <span class="n">behaves</span> <span class="k">as</span> <span class="k">if</span> <span class="n">awaitResponses</span><span class="p">(</span><span class="n">IdType</span><span class="p">,</span> <span class="n">timeout</span><span class="p">)</span>
<span class="n">were</span> <span class="n">invoked</span> <span class="n">on</span> <span class="nb">all</span> <span class="n">IDs</span><span class="o">.</span> <span class="n">The</span> <span class="n">returned</span> <span class="n">vector</span> <span class="n">contains</span>
<span class="n">a</span> <span class="n">vector</span> <span class="n">of</span> <span class="n">responses</span> <span class="n">per</span> <span class="n">ID</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">same</span> <span class="n">order</span> <span class="n">specified</span> <span class="n">by</span> <span class="n">the</span> <span class="n">requestIds</span><span class="o">.</span>
<span class="n">The</span> <span class="n">same</span> <span class="n">behaviour</span> <span class="k">as</span> <span class="n">awaitResponses</span><span class="p">(</span><span class="n">IdType</span><span class="p">,</span> <span class="n">timeout</span><span class="p">)</span> <span class="n">applies</span><span class="p">:</span>
<span class="o">*</span> <span class="n">Responses</span> <span class="n">may</span> <span class="n">be</span> <span class="n">empty</span><span class="o">.</span>
<span class="o">*</span> <span class="n">If</span> <span class="nb">all</span> <span class="n">responses</span> <span class="n">have</span> <span class="n">already</span> <span class="n">been</span> <span class="n">given</span> <span class="k">for</span> <span class="n">one</span> <span class="n">of</span> <span class="n">the</span> <span class="n">requestIds</span><span class="p">,</span>
<span class="n">then</span> <span class="n">this</span> <span class="n">method</span> <span class="n">will</span> <span class="n">hang</span> <span class="n">unless</span> <span class="n">a</span> <span class="n">timeout</span> <span class="ow">is</span> <span class="n">specified</span><span class="o">.</span>
</pre></div>
</div>
</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>requestIds</strong> – Ids requested </p></li>
<li><p><strong>timeout</strong> – The maximum time to wait for new responses </p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>A vector of vector of responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE">
<span id="_CPPv3NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE"></span><span id="_CPPv2NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE"></span><span id="tensorrt_llm::executor::Executor::getNumResponsesReady__std::optional:IdType:CRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a8870564de9e1b838e3a24e770de65e97"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getNumResponsesReady</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestId</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">nullopt</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Executor20getNumResponsesReadyERKNSt8optionalI6IdTypeEE" title="Link to this definition"></a><br /></dt>
<dd><p>Get the number of ready responses. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>requestId</strong> – An optional request id </p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>The number of ready responses </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor13cancelRequestE6IdType">
<span id="_CPPv3N12tensorrt_llm8executor8Executor13cancelRequestE6IdType"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor13cancelRequestE6IdType"></span><span id="tensorrt_llm::executor::Executor::cancelRequest__IdType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1ad157e5c600e0bcd1fb16695d573ec279"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cancelRequest</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">requestId</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor13cancelRequestE6IdType" title="Link to this definition"></a><br /></dt>
<dd><p>Cancel the request with provided request id. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>id</strong> – The request id for which to cancel the response </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor8shutdownEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor8shutdownEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor8shutdownEv"></span><span id="tensorrt_llm::executor::Executor::shutdown"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1abee0bdae43e18a83ada60f4e63eb72d5"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">shutdown</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor8shutdownEv" title="Link to this definition"></a><br /></dt>
<dd><p>Signals the server to shutdown This call is blocking. Only returns when all requests have terminated or timeout has been reached. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv"></span><span id="tensorrt_llm::executor::Executor::getLatestIterationStats"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a1fc213813dfe880c35767675737393b2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">deque</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLatestIterationStats</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor23getLatestIterationStatsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the per-iterations statistics computed since last call to getLatestIterationStats Contains at most iterStatsMaxIterations iterations. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p>Iteration stats </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv">
<span id="_CPPv3N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv"></span><span id="tensorrt_llm::executor::Executor::getLatestRequestStats"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a8ce8ded29df544b2b5d38b5cdb5308eb"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">deque</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getLatestRequestStats</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor21getLatestRequestStatsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the request stats of each iteration computed since last call to getLatestRequestStats Contains at most requestStatsMaxIterations iterations. </p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
<dd class="field-odd"><p><a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> stats grouped by iterations </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv">
<span id="_CPPv3NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv"></span><span id="tensorrt_llm::executor::Executor::canEnqueueRequestsC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a385dcd6e4c171cc8dcc6b7e327a8524b"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">canEnqueueRequests</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor8Executor18canEnqueueRequestsEv" title="Link to this definition"></a><br /></dt>
<dd><p>Indicates if the current process is allowed to enqueueRequests. </p>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8Executor5mImplE">
<span id="_CPPv3N12tensorrt_llm8executor8Executor5mImplE"></span><span id="_CPPv2N12tensorrt_llm8executor8Executor5mImplE"></span><span id="tensorrt_llm::executor::Executor::mImpl__std::unique_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Executor_1a4b7affd37a154513db81d23acb941801"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unique_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">Impl</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mImpl</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8Executor5mImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerializationE">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerializationE"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerializationE"></span><span id="tensorrt_llm::executor::JsonSerialization"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">JsonSerialization</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerializationE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;executor.h&gt;</em></div>
<p>Class with utility functions to serialize statistics to json string. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats"></span><span id="tensorrt_llm::executor::JsonSerialization::toJsonStr__IterationStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization_1abb7c3366de8460accfc20764a1c820c6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toJsonStr</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterationStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK14IterationStats" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to convert an iterationStats struct to a json serialized string. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration"></span><span id="tensorrt_llm::executor::JsonSerialization::toJsonStr__RequestStatsPerIterationCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization_1a8734846c416a9d9199d72c0fc6429374"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toJsonStr</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="tensorrt_llm::executor::RequestStatsPerIteration"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStatsPerIter</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK24RequestStatsPerIteration" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to convert a requestStatsPerIteration struct to a json serialized string. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats">
<span id="_CPPv3N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats"></span><span id="_CPPv2N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats"></span><span id="tensorrt_llm::executor::JsonSerialization::toJsonStr__RequestStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1JsonSerialization_1a7cbb6808bdbace0121a04044ec044854"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">toJsonStr</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">requestStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17JsonSerialization9toJsonStrERK12RequestStats" title="Link to this definition"></a><br /></dt>
<dd><p>Utility function to convert a requestStats struct to a json serialized string. </p>
</dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm3mpiE">
<span id="_CPPv3N12tensorrt_llm3mpiE"></span><span id="_CPPv2N12tensorrt_llm3mpiE"></span><span id="tensorrt_llm::mpi"></span><span class="target" id="namespacetensorrt__llm_1_1mpi"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mpi</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm3mpiE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="serialization-h">
<h2>serialization.h<a class="headerlink" href="#serialization-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13SerializationE">
<span id="_CPPv3N12tensorrt_llm8executor13SerializationE"></span><span id="_CPPv2N12tensorrt_llm8executor13SerializationE"></span><span id="tensorrt_llm::executor::Serialization"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Serialization</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13SerializationE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSamplingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a018e2c20ed62e05f0428c770990cf3a7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSamplingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeSamplingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__SamplingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab250085b9f35d5c2ca33e63241f4ffa5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14SamplingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__SamplingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2b0d1cd68b238760ff02f8a4740bead3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14SamplingConfigE" title="tensorrt_llm::executor::SamplingConfig"><span class="n"><span class="pre">SamplingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14SamplingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeOutputConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1afa9afb4d8f345960ce3419aa50a7aecb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeOutputConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeOutputConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__OutputConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a50563b0a86ded8ca3f7273d126ac7042"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12OutputConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__OutputConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aa51e9d72a4d69d15f2371d2eb8cbeba3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12OutputConfigE" title="tensorrt_llm::executor::OutputConfig"><span class="n"><span class="pre">OutputConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12OutputConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeExternalDraftTokensConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2bca807ceb86f58a385de9bdcc1bf481"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeExternalDraftTokensConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization36deserializeExternalDraftTokensConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ExternalDraftTokensConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1adbcd97e4e4d2822a1222fab34c3b3699"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK25ExternalDraftTokensConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ExternalDraftTokensConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a0c631d80dfe19e428b64e92e24ecbfc7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor25ExternalDraftTokensConfigE" title="tensorrt_llm::executor::ExternalDraftTokensConfig"><span class="n"><span class="pre">ExternalDraftTokensConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK25ExternalDraftTokensConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializePromptTuningConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9c20dc9bf0ec815a2fd91243e79f82a1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializePromptTuningConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializePromptTuningConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__PromptTuningConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aeffa6dbca81617e4c8f2f151402de0aa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18PromptTuningConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__PromptTuningConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a64114e901f6976ad2ede341a4ce46623"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18PromptTuningConfigE" title="tensorrt_llm::executor::PromptTuningConfig"><span class="n"><span class="pre">PromptTuningConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18PromptTuningConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeLoraConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab6ae74552a93ff397c5af265c344fe56"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeLoraConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization21deserializeLoraConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__LoraConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aab0cf47956bc7bb1ee1452aa90edb6de"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK10LoraConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__LoraConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a70a62ef2cdbc5a65dc3b61e4052a8133"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10LoraConfigE" title="tensorrt_llm::executor::LoraConfig"><span class="n"><span class="pre">LoraConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">config</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK10LoraConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeRequest__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a52ea6afb73b932c514d8bf2fbacb5df2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeRequest</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization18deserializeRequestERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__RequestCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a51b1f66893f945e1d48ce4f466ba1010"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK7RequestRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__RequestCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a15092605072969164a22559fcf61f6c1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor7RequestE" title="tensorrt_llm::executor::Request"><span class="n"><span class="pre">Request</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">request</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK7Request" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeTensor__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ab32ab17b2c60fba0ee29097c2ea7e055"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeTensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeTensorERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__TensorCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8fe83cad52a85278ba6ff00c542a9214"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6TensorRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a85dbca4cdebadd45d7329329fcf656c0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeResult__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a258edbaa27d4bc82e5919f921aaff5b3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeResult</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeResultERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ResultCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae2bc066744d82a2457a974478a92d24d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">result</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK6ResultRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ResultCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1acf908575acc37c7e106488e59f8aa4ff"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6ResultE" title="tensorrt_llm::executor::Result"><span class="n"><span class="pre">Result</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">result</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK6Result" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeResponse__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae5a8b5e299fb2231fc41881b2fd12b31"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeResponse</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization19deserializeResponseERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ResponseCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a1dc5a101175bb9de92efa650e6d20e68"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">response</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK8ResponseRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ResponseCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1accb642acb602af27140cce717b5b18e8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">response</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK8Response" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeResponses__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9155d1000c9ee0663b5cfbc9c385d740"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeResponses</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeResponsesERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE"></span><span id="tensorrt_llm::executor::Serialization::serialize__std::vector:Response:CR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad4ee918c7332771d2770fa4df15e7a7c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8ResponseE" title="tensorrt_llm::executor::Response"><span class="n"><span class="pre">Response</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">responses</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERKNSt6vectorI8ResponseEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeKvCacheConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a93df88f47d0cbfa681af463ed1b680d8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeKvCacheConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization24deserializeKvCacheConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__KvCacheConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8e27bfce7f4b56448cdb1e40596ad5b6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK13KvCacheConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__KvCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad457e1d524480c49cc90ac55aeda1943"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13KvCacheConfigE" title="tensorrt_llm::executor::KvCacheConfig"><span class="n"><span class="pre">KvCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK13KvCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeSchedulerConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5721cef6b72c2a079b5dbe94a7f9249e"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeSchedulerConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializeSchedulerConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__SchedulerConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a75d2e0606543f60ed2bf6bbaf00456f4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15SchedulerConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__SchedulerConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1af3642582eaeb939ba265089c2fee753a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15SchedulerConfigE" title="tensorrt_llm::executor::SchedulerConfig"><span class="n"><span class="pre">SchedulerConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">schedulerConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15SchedulerConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeParallelConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8ddf1b2798dbf98d3eac0759c4646360"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeParallelConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeParallelConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ParallelConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a352018daa6065ab5d7f8d40df594f790"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">parallelConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ParallelConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ParallelConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a903b5bc4b669a8e8cfdff9206512969a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ParallelConfigE" title="tensorrt_llm::executor::ParallelConfig"><span class="n"><span class="pre">ParallelConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">parallelConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ParallelConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializePeftCacheConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a36ab38bb0f5fe55b9829c8177c93e91f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializePeftCacheConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization26deserializePeftCacheConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__PeftCacheConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a28c920c3cbb2c7f97678a0ed2fd704cb"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK15PeftCacheConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__PeftCacheConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae6b46926ad9c8754e7cd89c1b9c8bd36"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor15PeftCacheConfigE" title="tensorrt_llm::executor::PeftCacheConfig"><span class="n"><span class="pre">PeftCacheConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">peftCacheConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK15PeftCacheConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeOrchestratorConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ae025cd58097d96dc74cb1acd6207325d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeOrchestratorConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization29deserializeOrchestratorConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__OrchestratorConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a7864913d0c2362e972654c2a5613b566"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK18OrchestratorConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__OrchestratorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a255cb93a98ccdf748db37f8b492520d3"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor18OrchestratorConfigE" title="tensorrt_llm::executor::OrchestratorConfig"><span class="n"><span class="pre">OrchestratorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">orchestratorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK18OrchestratorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDecodingMode__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6793f998581467d3540e2ed35167be15"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDecodingMode</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeDecodingModeERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DecodingModeCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a0b3f4dd50c3174e5522cd945a55416a8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingMode</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12DecodingModeRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DecodingModeCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a68100e2eb882d29677a4c9ed45cfd7a8"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingMode</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12DecodingMode" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeLookaheadDecodingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a09e3d04d4394b505c41a2f88d2395de2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeLookaheadDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization34deserializeLookaheadDecodingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__LookaheadDecodingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a7f7d5716bc415282ad7d870cee8a0af2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK23LookaheadDecodingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__LookaheadDecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5dce85091047f4b569a106630ddd0840"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23LookaheadDecodingConfigE" title="tensorrt_llm::executor::LookaheadDecodingConfig"><span class="n"><span class="pre">LookaheadDecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">lookaheadDecodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK23LookaheadDecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeDecodingConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a60e192f644c0e8693f0a3b12d6bb60e0"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeDecodingConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeDecodingConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__DecodingConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad1a7a61b52c5d823406a5a2bb05edb21"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14DecodingConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__DecodingConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8b8d2707b981fbb788a24a31c1170a72"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14DecodingConfigE" title="tensorrt_llm::executor::DecodingConfig"><span class="n"><span class="pre">DecodingConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">decodingConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14DecodingConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeExecutorConfig__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a3bdc71d1f61ffbe8192eec4c69a6f863"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeExecutorConfig</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeExecutorConfigERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__ExecutorConfigCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8d327c320f459d69fd6561a420558674"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14ExecutorConfigRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__ExecutorConfigCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1aeabfd8c6625ad85aed6ef2cb72f8cd66"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14ExecutorConfigE" title="tensorrt_llm::executor::ExecutorConfig"><span class="n"><span class="pre">ExecutorConfig</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">executorConfig</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14ExecutorConfig" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeKvCacheStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a00364ba63856dbd99f89ae1e45770ffa"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeKvCacheStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization23deserializeKvCacheStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__KvCacheStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a92003eb85d2979660e29056b4f81cf0a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK12KvCacheStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__KvCacheStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a5e3c1533994cd82ce444b0c632bb6fbe"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">kvCacheStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK12KvCacheStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeStaticBatchingStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad66b899cda69588f655b08dd36f15925"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeStaticBatchingStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization30deserializeStaticBatchingStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__StaticBatchingStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a1187b7a88a0e5deb38ff8e593372e984"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">staticBatchingStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK19StaticBatchingStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__StaticBatchingStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6fe6208a4f78db241e6d9c6b2afd24e5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">staticBatchingStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK19StaticBatchingStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeInflightBatchingStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a6ddddf159c364fc1af5d7ba7a1a5c9cf"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeInflightBatchingStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization32deserializeInflightBatchingStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__InflightBatchingStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a1e571817bc0585695894442bd4697fc6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inflightBatchingStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK21InflightBatchingStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__InflightBatchingStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1adfd862d5d685d90dfbec4e742c340cd7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">inflightBatchingStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK21InflightBatchingStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE"></span><span id="tensorrt_llm::executor::Serialization::deserializeIterationStats__std::vector:c:R"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9797c0d1af10c396b36f548de7d2e8e2"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeIterationStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">buffer</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt6vectorIcEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeIterationStats__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ad1bbfdc68d6c6d0a8342317fac32bd93"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeIterationStats</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization25deserializeIterationStatsERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE"></span><span id="tensorrt_llm::executor::Serialization::serialize__IterationStatsCR.osR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a2ed4d91cf95dfd30083fa0154f8d6243"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStats</span></span>, <span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStatsRNSt7ostreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats"></span><span id="tensorrt_llm::executor::Serialization::serialize__IterationStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a8be0c1ff1c9f3dd71a125f3ed23777d9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">char</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serialize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization9serializeERK14IterationStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats"></span><span id="tensorrt_llm::executor::Serialization::serializedSize__IterationStatsCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a24a30c0a00c745499a73d7f754c9e67a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">serializedSize</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="tensorrt_llm::executor::IterationStats"><span class="n"><span class="pre">IterationStats</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">iterStats</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization14serializedSizeERK14IterationStats" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeString__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1ac520ac5f4cc9ec6f5b6bf53ab5b7da8f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeString</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization17deserializeStringERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeBool__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a19d98b5105004255595bfeecd7cecb4a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeBool</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization15deserializeBoolERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE">
<span id="_CPPv3N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE"></span><span id="_CPPv2N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE"></span><span id="tensorrt_llm::executor::Serialization::deserializeModelType__isR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Serialization_1a9944e16a2c2b64e441b36fa72afb2e6b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="tensorrt_llm::executor::ModelType"><span class="n"><span class="pre">ModelType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">deserializeModelType</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">istream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">is</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13Serialization20deserializeModelTypeERNSt7istreamE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
</dd></dl>
</section>
<section id="tensor-h">
<h2>tensor.h<a class="headerlink" href="#tensor-h" title="Link to this heading"></a></h2>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5ShapeE">
<span id="_CPPv3N12tensorrt_llm8executor5ShapeE"></span><span id="_CPPv2N12tensorrt_llm8executor5ShapeE"></span><span id="tensorrt_llm::executor::Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="w"> </span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="k"><span class="pre">public</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ArrayView</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detailE" title="tensorrt_llm::executor::detail"><span class="n"><span class="pre">detail</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E" title="tensorrt_llm::executor::detail::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape4BaseE">
<span id="_CPPv3N12tensorrt_llm8executor5Shape4BaseE"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape4BaseE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1abea4be68b98825ce00bd278dd61492ac"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Base</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">common</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ArrayView</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detailE" title="tensorrt_llm::executor::detail"><span class="n"><span class="pre">detail</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E" title="tensorrt_llm::executor::detail::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape9DimType64E">
<span id="_CPPv3N12tensorrt_llm8executor5Shape9DimType64E"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape9DimType64E"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1a83c46d18b03c349f865a290209be5528"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DimType64</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">remove_cv_t</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE" title="tensorrt_llm::executor::Shape::Base"><span class="n"><span class="pre">Base</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">value_type</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape5ShapeEv">
<span id="_CPPv3N12tensorrt_llm8executor5Shape5ShapeEv"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape5ShapeEv"></span><span id="tensorrt_llm::executor::Shape::Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1a476b2b6a441e9fcbb8cad03b354b6681"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE">
<span id="_CPPv3N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE"></span><span id="tensorrt_llm::executor::Shape::Shape__DimType64CP.Base::size_type"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1aadbc8651b94baa75a3e2eb2cff889293"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E" title="tensorrt_llm::executor::Shape::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape4BaseE" title="tensorrt_llm::executor::Shape::Base"><span class="n"><span class="pre">Base</span></span></a><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_type</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">size</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeEPK9DimType64N4Base9size_typeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE">
<span id="_CPPv3N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE"></span><span id="_CPPv2N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE"></span><span id="tensorrt_llm::executor::Shape::Shape__std::initializer_list:DimType64:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Shape_1a9211bd067fe9f04c94a49decf4b812de"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Shape</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">initializer_list</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5Shape9DimType64E" title="tensorrt_llm::executor::Shape::DimType64"><span class="n"><span class="pre">DimType64</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">dims</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor5Shape5ShapeENSt16initializer_listI9DimType64EE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensorE">
<span id="_CPPv3N12tensorrt_llm8executor6TensorE"></span><span id="_CPPv2N12tensorrt_llm8executor6TensorE"></span><span id="tensorrt_llm::executor::Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor13CudaStreamPtrE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor13CudaStreamPtrE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab192218140e150441c3809208318577d"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CudaStreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToCpu__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a50aa2db7bc31318e493941918a304f6a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToCpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToCpuEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToPinned__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a97a17d0f122678e4269d578f8bbcc057"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor12copyToPinnedEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToPooledPinned__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a772e702ed0364b8fe922b1488e06f42d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToPooledPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor18copyToPooledPinnedEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToManaged__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a82d22baf219b3cf39e0ad7ea010e5457"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToManaged</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13copyToManagedEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE"></span><span id="tensorrt_llm::executor::Tensor::copyToGpu__Tensor::CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1aab06656d158bf52a97fdb554273bedd7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyToGpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor9copyToGpuEN6Tensor13CudaStreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorEv">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorEv"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorEv"></span><span id="tensorrt_llm::executor::Tensor::Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ae79dbd3ecc8694d366f5e2b67556a8cd"></span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensorD0Ev">
<span id="_CPPv3N12tensorrt_llm8executor6TensorD0Ev"></span><span id="_CPPv2N12tensorrt_llm8executor6TensorD0Ev"></span><span id="tensorrt_llm::executor::Tensor::~Tensor"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a28b3397651c85c4065e25d8d10d1c103"></span><span class="sig-name descname"><span class="n"><span class="pre">~Tensor</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensorD0Ev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::Tensor__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a785597fe1ac8b56286d4ad836ec70035"></span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor" title="tensorrt_llm::executor::Tensor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorERR6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorERR6Tensor"></span><span id="tensorrt_llm::executor::Tensor::Tensor__TensorRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ae40ad8a4bf115014f6ed52d674b07461"></span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor" title="tensorrt_llm::executor::Tensor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorERR6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensoraSERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6TensoraSERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6TensoraSERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::assign-operator__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a10ba81a156ca4ec842f982bc7cc5f5ef"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6TensoraSERR6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6TensoraSERR6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6TensoraSERR6Tensor"></span><span id="tensorrt_llm::executor::Tensor::assign-operator__TensorRR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a51db02740c598df782229b073de130d1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">noexcept</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">default</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6TensoraSERR6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7getDataEv">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="tensorrt_llm::executor::Tensor::getData"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1adf53933fa4c01c0e4cd1cc2df7526afb"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getData</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7getDataEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor7getDataEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor7getDataEv"></span><span id="tensorrt_llm::executor::Tensor::getDataC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab600be03e97cdb89d015dbf5611068fd"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="sig-name descname"><span class="n"><span class="pre">getData</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getDataEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns a pointer to underlying array. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor11getDataTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor11getDataTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor11getDataTypeEv"></span><span id="tensorrt_llm::executor::Tensor::getDataTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a745614bf656424a3576690a25b5afbd4"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getDataType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor11getDataTypeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the data type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv"></span><span id="tensorrt_llm::executor::Tensor::getMemoryTypeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab60d4a2939254f259d6354714bb3396a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10MemoryTypeE" title="tensorrt_llm::executor::MemoryType"><span class="n"><span class="pre">MemoryType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getMemoryType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor13getMemoryTypeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the memory type of the buffer. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor8getShapeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor8getShapeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor8getShapeEv"></span><span id="tensorrt_llm::executor::Tensor::getShapeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab6f07c9a28710562e4366187b6d465cb"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getShape</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor8getShapeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the tensor dimensions. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor7getSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor7getSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor7getSizeEv"></span><span id="tensorrt_llm::executor::Tensor::getSizeC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a65ca052e56fa4b4b819456678a80cb82"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor7getSizeEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the number of elements in the tensor. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv"></span><span id="tensorrt_llm::executor::Tensor::getSizeInBytesC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ae05d5b4735c1aeb2536903d51c2dc2fd"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getSizeInBytes</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor14getSizeInBytesEv" title="Link to this definition"></a><br /></dt>
<dd><p>Returns the size of the tensor in bytes. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr"></span><span id="tensorrt_llm::executor::Tensor::setZero__CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ad0cfa186ced1eb92039987e745aa5174"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setZero</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setZeroE13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Set the entire memory to zero. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>stream</strong> – Must be a valid CUDA stream if the memory type is GPU. </p>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr"></span><span id="tensorrt_llm::executor::Tensor::setFrom__TensorCR.CudaStreamPtr"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ab74daacf178945a17ecd8552ad6ea733"></span><span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setFrom</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">nullptr</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7setFromERK6Tensor13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd><p>Copy the data and shape from another tensor. </p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>other</strong> – A tensor to copy from. </p></li>
<li><p><strong>stream</strong> – Must be a valid CUDA stream if the memory type is GPU. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6TensorcvbEv">
<span id="_CPPv3NK12tensorrt_llm8executor6TensorcvbEv"></span><span id="_CPPv2NK12tensorrt_llm8executor6TensorcvbEv"></span><span id="tensorrt_llm::executor::Tensor::castto-b-operatorC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a6c8f0a832643e66193ac9246423769ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6TensorcvbEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6TensoreqERK6Tensor">
<span id="_CPPv3NK12tensorrt_llm8executor6TensoreqERK6Tensor"></span><span id="_CPPv2NK12tensorrt_llm8executor6TensoreqERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::eq-operator__TensorCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a80ff07f5b580df282b96a573c8e5bd5a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">rhs</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6TensoreqERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6TensorneERK6Tensor">
<span id="_CPPv3NK12tensorrt_llm8executor6TensorneERK6Tensor"></span><span id="_CPPv2NK12tensorrt_llm8executor6TensorneERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::neq-operator__TensorCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a35337e466b30dab925a4e895406214ac"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">!=</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">rhs</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6TensorneERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::cpu__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ac116935be582b663662c6f39c7b774a5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor3cpuE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a cpu tensor with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor3cpuE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor3cpuE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor3cpuE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a7ddcb397b6da797ca7825e93841abf19"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3cpuE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::pinned__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1aced97f5afb664a7dfc416a02b587462a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6pinnedE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a cpu tensor in pinned memory with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor6pinnedE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor6pinnedE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor6pinnedE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1ada1a99697b667dd46221f73a9a9ee7b9"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor6pinnedE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::pooledPinned__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a7ba670c87ea79d7e66d562c22d7821ca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pooledPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor12pooledPinnedE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a cpu tensor in pooled pinned memory with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a66ac66e737e5481c8f1640f15ce613fc"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pooledPinned</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor12pooledPinnedE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape"></span><span id="tensorrt_llm::executor::Tensor::managed__DataType.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a7731d566daed52054eccabfe3de81e1a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">managed</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7managedE8DataType5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a tensor in managed memory (UVM) with the given shape and data type.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor7managedE6Tensor5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor7managedE5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor7managedE5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1aa95d4b6ccfcc20b47fb6a0145073c57d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">managed</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor7managedE6Tensor5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape"></span><span id="tensorrt_llm::executor::Tensor::gpu__DataType.CudaStreamPtr.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a3fe1404ac1edb89baebaeb852817823d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor3gpuE8DataType13CudaStreamPtr5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Allocate a gpu tensor with the given shape and data type on a particular cuda stream.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor3gpuE6Tensor13CudaStreamPtr5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor3gpuE13CudaStreamPtr5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor3gpuE13CudaStreamPtr5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a0bbc0df61a4c8c3187278b66f38f41fe"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpu</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor3gpuE6Tensor13CudaStreamPtr5Shape" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape"></span><span id="tensorrt_llm::executor::Tensor::of__DataType.voidP.Shape"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a53fce3c66408ad1da79dc838ea9a1921"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">of</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">dataType</span></span>, <span class="kt"><span class="pre">void</span></span><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor2ofE8DataTypePv5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Wrap a data pointer into a tensor without taking ownership.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor2ofEP1T5Shape"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor2ofEP1T5Shape"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1abb953f082c26af69c559128e3e30b8ca"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">of</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape" title="tensorrt_llm::executor::Tensor::of::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">*</span></span><span class="n sig-param"><span class="pre">data</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor5ShapeE" title="tensorrt_llm::executor::Shape"><span class="n"><span class="pre">Shape</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">shape</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorP1T5Shape" title="Link to this definition"></a><br /></dt>
<dd><p>Wrap a data pointer into a tensor without taking ownership.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor2ofER1T"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor2ofER1T"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a367caf7b52fc7ccef5fe8e879376743b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">of</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T" title="tensorrt_llm::executor::Tensor::of::T"><span class="n"><span class="pre">T</span></span></a><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">data</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor2ofE6TensorR1T" title="Link to this definition"></a><br /></dt>
<dd><p>Wrap any container into a tensor without taking ownership.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>shape</strong> – The shape of the tensor. </p></li>
<li><p><strong>dataType</strong> – The data type of the tensor. </p></li>
<li><p><strong>stream</strong> – Specifies the CUDA stream on which to allocate the tensor for GPU memory. </p></li>
</ul>
</dd>
</dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-types">Private Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor4ImplE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor4ImplE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor4ImplE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a499dd2cff3f5f1ebd0a2a508462af729"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Impl</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="tensorrt_llm::executor::Tensor::Tensor__std::shared_ptr:runtime::ITensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1abaff3fb79d1193b805d9ed9182f9482f"></span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Tensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6TensorENSt10shared_ptrIN7runtime7ITensorEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr">
<span id="_CPPv3NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr"></span><span id="_CPPv2NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr"></span><span id="tensorrt_llm::executor::Tensor::copyTo__std::shared_ptr:Impl:.CudaStreamPtrC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1adce5c2600b80e6aac03d72f62c442eaa"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">copyTo</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE" title="tensorrt_llm::executor::Tensor::Impl"><span class="n"><span class="pre">Impl</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor13CudaStreamPtrE" title="tensorrt_llm::executor::Tensor::CudaStreamPtr"><span class="n"><span class="pre">CudaStreamPtr</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">stream</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor6Tensor6copyToENSt10shared_ptrI4ImplEE13CudaStreamPtr" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor7mTensorE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor7mTensorE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor7mTensorE"></span><span id="tensorrt_llm::executor::Tensor::mTensor__std::shared_ptr:Impl:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1af5858fa2abb457b83fc8aa0cfa0392ce"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6Tensor4ImplE" title="tensorrt_llm::executor::Tensor::Impl"><span class="n"><span class="pre">Impl</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mTensor</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor7mTensorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-functions">Private Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeE8DataTypev">
<span id="_CPPv3I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeEv"></span><span id="_CPPv2I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeEv"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a323f2561238fa6191bf4b18b4916a464"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getRuntimeType</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor6Tensor14getRuntimeTypeE8DataTypev" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-friends">Friends</p>
<dl>
<dt class="sig sig-object cpp">
<em class="property"><span class="pre">friend</span> <span class="pre">class</span></em> <span class="pre">Serialization</span></dt>
</dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor"></span><span id="tensorrt_llm::executor::Tensor::detail::toITensor__TensorCR"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a03d81c30ba47afa0f46d55063ba2f74f"></span><span class="k"><span class="pre">friend</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">toITensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9toITensorERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE">
<span id="_CPPv3N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="_CPPv2N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="tensorrt_llm::executor::Tensor::detail::ofITensor__std::shared_ptr:runtime::ITensor:"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1Tensor_1a8e986804106e3fe17218fbeebcb2659c"></span><span class="k"><span class="pre">friend</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ofITensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6Tensor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detailE">
<span id="_CPPv3N12tensorrt_llm8executor6detailE"></span><span id="_CPPv2N12tensorrt_llm8executor6detailE"></span><span id="tensorrt_llm::executor::detail"></span><span class="target" id="namespacetensorrt__llm_1_1executor_1_1detail"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">detail</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detailE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detail9DimType64E">
<span id="_CPPv3N12tensorrt_llm8executor6detail9DimType64E"></span><span id="_CPPv2N12tensorrt_llm8executor6detail9DimType64E"></span><span class="target" id="tensor_8h_1aaa88c014789556bcf4f890ef4f91842b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DimType64</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">int64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detail9DimType64E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detail9toITensorERK6Tensor">
<span id="_CPPv3N12tensorrt_llm8executor6detail9toITensorERK6Tensor"></span><span id="_CPPv2N12tensorrt_llm8executor6detail9toITensorERK6Tensor"></span><span id="tensorrt_llm::executor::detail::toITensor__TensorCR"></span><span class="target" id="tensor_8h_1a973860a4bc7eff6a999b21d12b709d82"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="n"><span class="pre">toITensor</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detail9toITensorERK6Tensor" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE">
<span id="_CPPv3N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="_CPPv2N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE"></span><span id="tensorrt_llm::executor::detail::ofITensor__std::shared_ptr:runtime::ITensor:"></span><span class="target" id="tensor_8h_1acb29416de14060f4546f8653af9220e8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ofITensor</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime7ITensorE" title="tensorrt_llm::runtime::ITensor"><span class="n"><span class="pre">ITensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">tensor</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6detail9ofITensorENSt10shared_ptrIN7runtime7ITensorEEE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm7runtimeE">
<span id="_CPPv3N12tensorrt_llm7runtimeE"></span><span id="_CPPv2N12tensorrt_llm7runtimeE"></span><span id="tensorrt_llm::runtime"></span><span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm7runtimeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
</section>
<section id="types-h">
<h2>types.h<a class="headerlink" href="#types-h" title="Link to this heading"></a></h2>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt6int8_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::int8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4_1aaed76141f49476650f91cfbec3367e13"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E" title="tensorrt_llm::executor::DataType::kINT8"><span class="n"><span class="pre">kINT8</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt6int8_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int32_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::int32_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4_1a506e37fcee8102b90f320257e12ec485"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E" title="tensorrt_llm::executor::DataType::kINT32"><span class="n"><span class="pre">kINT32</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int32_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int64_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7int64_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::int64_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4_1ae1e174a531a0abf5c31ed44525da252d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="tensorrt_llm::executor::DataType::kINT64"><span class="n"><span class="pre">kINT64</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7int64_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint8_t</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:std::uint8_t:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4_1a36e64efaf2f35f9e9d97c62fb4e74f84"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E" title="tensorrt_llm::executor::DataType::kUINT8"><span class="n"><span class="pre">kUINT8</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsINSt7uint8_tEE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tensorrt_llm</span></span></span><br /></dt>
<dd><dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1executor"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">executor</span></span></span><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-typedefs">Typedefs</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9TensorPtrE">
<span id="_CPPv3N12tensorrt_llm8executor9TensorPtrE"></span><span id="_CPPv2N12tensorrt_llm8executor9TensorPtrE"></span><span class="target" id="types_8h_1a32a3846eb7d506ec2f4699f052f54dda"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TensorPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9TensorPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10SizeType32E">
<span id="_CPPv3N12tensorrt_llm8executor10SizeType32E"></span><span id="_CPPv2N12tensorrt_llm8executor10SizeType32E"></span><span class="target" id="types_8h_1ad818c2e487265ea3ec0ddd760b768085"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">SizeType32</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9FloatTypeE">
<span id="_CPPv3N12tensorrt_llm8executor9FloatTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor9FloatTypeE"></span><span class="target" id="types_8h_1a48053cc72a5a67b3c19c817fb963ecea"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">FloatType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="kt"><span class="pre">float</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11TokenIdTypeE">
<span id="_CPPv3N12tensorrt_llm8executor11TokenIdTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor11TokenIdTypeE"></span><span class="target" id="types_8h_1a5658d78655723ba6fac682b945cc6788"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TokenIdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">int32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9VecTokensE">
<span id="_CPPv3N12tensorrt_llm8executor9VecTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor9VecTokensE"></span><span class="target" id="types_8h_1a3f6e6a65b5e81747e30820e27d184e91"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">VecTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor11TokenIdTypeE" title="tensorrt_llm::executor::TokenIdType"><span class="n"><span class="pre">TokenIdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10BeamTokensE">
<span id="_CPPv3N12tensorrt_llm8executor10BeamTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor10BeamTokensE"></span><span class="target" id="types_8h_1afc3b526e44121eda1f3344e1d611f688"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BeamTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9VecTokensE" title="tensorrt_llm::executor::VecTokens"><span class="n"><span class="pre">VecTokens</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor6IdTypeE">
<span id="_CPPv3N12tensorrt_llm8executor6IdTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor6IdTypeE"></span><span class="target" id="types_8h_1ab9563a6f39b5785365973b84532f9353"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IdType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13IterationTypeE">
<span id="_CPPv3N12tensorrt_llm8executor13IterationTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor13IterationTypeE"></span><span class="target" id="types_8h_1ad5297846b0ec6db8536e14e970e0e09c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IterationType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE">
<span id="_CPPv3N12tensorrt_llm8executor14RandomSeedTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor14RandomSeedTypeE"></span><span class="target" id="types_8h_1aa7776f1267bf68fcfd4228fb0cc38bfc"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RandomSeedType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">uint64_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14RandomSeedTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor11VecLogProbsE">
<span id="_CPPv3N12tensorrt_llm8executor11VecLogProbsE"></span><span id="_CPPv2N12tensorrt_llm8executor11VecLogProbsE"></span><span class="target" id="types_8h_1af438a899c644096dfd518bee78fad2dd"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">VecLogProbs</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9FloatTypeE" title="tensorrt_llm::executor::FloatType"><span class="n"><span class="pre">FloatType</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor11VecLogProbsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9StreamPtrE">
<span id="_CPPv3N12tensorrt_llm8executor9StreamPtrE"></span><span id="_CPPv2N12tensorrt_llm8executor9StreamPtrE"></span><span class="target" id="types_8h_1a37eb662d6bef7e3702dac46671c3f543"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StreamPtr</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">shared_ptr</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv412tensorrt_llm" title="tensorrt_llm"><span class="n"><span class="pre">tensorrt_llm</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm7runtimeE" title="tensorrt_llm::runtime"><span class="n"><span class="pre">runtime</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="runtime.html#_CPPv4N12tensorrt_llm7runtime10CudaStreamE" title="tensorrt_llm::runtime::CudaStream"><span class="n"><span class="pre">CudaStream</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE">
<span id="_CPPv3N12tensorrt_llm8executor19LogitsPostProcessorE"></span><span id="_CPPv2N12tensorrt_llm8executor19LogitsPostProcessorE"></span><span class="target" id="types_8h_1a2c6a140b6f50b8761e77c1fd7a57cfed"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessor</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="tensorrt_llm::executor::BeamTokens"><span class="n"><span class="pre">BeamTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE" title="tensorrt_llm::executor::StreamPtr"><span class="n"><span class="pre">StreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE">
<span id="_CPPv3N12tensorrt_llm8executor22LogitsPostProcessorMapE"></span><span id="_CPPv2N12tensorrt_llm8executor22LogitsPostProcessorMapE"></span><span class="target" id="types_8h_1ad609221eb06a044bdfa9afc31ac4b69b"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessorMap</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">unordered_map</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19LogitsPostProcessorE" title="tensorrt_llm::executor::LogitsPostProcessor"><span class="n"><span class="pre">LogitsPostProcessor</span></span></a><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor22LogitsPostProcessorMapE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE">
<span id="_CPPv3N12tensorrt_llm8executor26LogitsPostProcessorBatchedE"></span><span id="_CPPv2N12tensorrt_llm8executor26LogitsPostProcessorBatchedE"></span><span class="target" id="types_8h_1ad5232a896a7929593c229f1eec00c5e2"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">LogitsPostProcessorBatched</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">function</span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">void</span></span><span class="p"><span class="pre">(</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6TensorE" title="tensorrt_llm::executor::Tensor"><span class="n"><span class="pre">Tensor</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">reference_wrapper</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10BeamTokensE" title="tensorrt_llm::executor::BeamTokens"><span class="n"><span class="pre">BeamTokens</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor9StreamPtrE" title="tensorrt_llm::executor::StreamPtr"><span class="n"><span class="pre">StreamPtr</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="p"><span class="pre">&amp;</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor26LogitsPostProcessorBatchedE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor13MedusaChoicesE">
<span id="_CPPv3N12tensorrt_llm8executor13MedusaChoicesE"></span><span id="_CPPv2N12tensorrt_llm8executor13MedusaChoicesE"></span><span class="target" id="types_8h_1a6ee0e4eac427f8b44cd02cad361df400"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MedusaChoices</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor13MedusaChoicesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-enums">Enums</p>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataTypeE">
<span id="_CPPv3N12tensorrt_llm8executor8DataTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor8DataTypeE"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96e"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DataType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kBOOLE"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kBOOLE"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea65d294ade613161e5ea557e935133213"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBOOL</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType6kUINT8E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType6kUINT8E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea159244e0ab6cea87daa7eaa8f6e19937"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUINT8</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kINT8E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kINT8E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kINT8E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea69c1a4a69db0e50820cf63122f90ad09"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINT8</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType6kINT32E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType6kINT32E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType6kINT32E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96eabd073fcbb15020b25a70e2cd95f9f4a9"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINT32</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType6kINT64E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType6kINT64E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType6kINT64E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea223b86006bb063f2a58200621e0656d5"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINT64</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kBF16E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kBF16E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kBF16E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea7b2caf2ed7b5f4f177fe3207cd198791"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBF16</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBF16E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType4kFP8E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType4kFP8E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType4kFP8E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea7397615c6bee5b62289fc7cceb82fbf7"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFP8</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType4kFP8E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kFP16E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kFP16E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kFP16E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea56e4ef5e47a48568bd24c4e0aaabcead"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFP16</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP16E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType5kFP32E">
<span id="_CPPv3N12tensorrt_llm8executor8DataType5kFP32E"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType5kFP32E"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea6c6463a7e81555667de0d7b49101a701"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFP32</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP32E" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor8DataType8kUNKNOWNE">
<span id="_CPPv3N12tensorrt_llm8executor8DataType8kUNKNOWNE"></span><span id="_CPPv2N12tensorrt_llm8executor8DataType8kUNKNOWNE"></span><span class="target" id="types_8h_1ad496a8556f8507f89bea4b8b7a7fc96ea16bb40b9102367393ecf8213078d1c6e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUNKNOWN</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor8DataType8kUNKNOWNE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryTypeE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryTypeE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aab"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">MemoryType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType4kCPUE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType4kCPUE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType4kCPUE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba2ce310327f474afc9f6774faa2f57903"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kCPUE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aabaa12edb33c24c693b60e591681cfb1e66"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCPU_PINNED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType11kCPU_PINNEDE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType4kGPUE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType4kGPUE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType4kGPUE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba37ae524b76587efa776affdc5cdf2ac1"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGPU</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kGPUE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType4kUVME">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType4kUVME"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType4kUVME"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba7d7de0bd70e1276255098b25010bdeb6"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUVM</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType4kUVME" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10MemoryType8kUNKNOWNE">
<span id="_CPPv3N12tensorrt_llm8executor10MemoryType8kUNKNOWNE"></span><span id="_CPPv2N12tensorrt_llm8executor10MemoryType8kUNKNOWNE"></span><span class="target" id="types_8h_1a8399248ba3babb51a2d0802a17e52aaba16bb40b9102367393ecf8213078d1c6e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUNKNOWN</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10MemoryType8kUNKNOWNE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelTypeE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelTypeE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ModelType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46a577f62bbac6fac26ae5fcbfbd638405b"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kDECODER_ONLY</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kDECODER_ONLYE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46ab9f0d192962e27b51920b525cfc0c8a6"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kENCODER_ONLY</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelType13kENCODER_ONLYE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE">
<span id="_CPPv3N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE"></span><span id="_CPPv2N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE"></span><span class="target" id="types_8h_1ac442c56f38328a24c7fb20bdd7543c46a9faae00b617c7bb72d509e32454d1758"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kENCODER_DECODER</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor9ModelType16kENCODER_DECODERE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12BatchingTypeE">
<span id="_CPPv3N12tensorrt_llm8executor12BatchingTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor12BatchingTypeE"></span><span class="target" id="types_8h_1aca90eaee2e69172d309f77a581b824e3"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BatchingType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12BatchingTypeE" title="Link to this definition"></a><br /></dt>
<dd><p>The batching type. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12BatchingType7kSTATICE">
<span id="_CPPv3N12tensorrt_llm8executor12BatchingType7kSTATICE"></span><span id="_CPPv2N12tensorrt_llm8executor12BatchingType7kSTATICE"></span><span class="target" id="types_8h_1aca90eaee2e69172d309f77a581b824e3a201894ba187a596f534f6406676f96bf"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kSTATIC</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12BatchingType7kSTATICE" title="Link to this definition"></a><br /></dt>
<dd><p>STATIC refers to the traditional batching scheme with a batch of requests running in lockstep until the full generation for all of them is complete. Requests in a batch are all padded up to the maximum input and output sequence length of any member of the batch. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE">
<span id="_CPPv3N12tensorrt_llm8executor12BatchingType9kINFLIGHTE"></span><span id="_CPPv2N12tensorrt_llm8executor12BatchingType9kINFLIGHTE"></span><span class="target" id="types_8h_1aca90eaee2e69172d309f77a581b824e3af9f499a6c3b205667d7f5ddba6bf4c02"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kINFLIGHT</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12BatchingType9kINFLIGHTE" title="Link to this definition"></a><br /></dt>
<dd><p>INFLIGHT refers to a scheme where newly arrived requests are dynamically incorporated into the batch under execution, and requests are returned as soon as the end condition is met without any padding. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicyE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p>The policy used to select the subset of available requests in each iteration of the executor generation loop. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428ae7df1f3a699ee2bfbe4328f94e825b7e"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMAX_UTILIZATION</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy16kMAX_UTILIZATIONE" title="Link to this definition"></a><br /></dt>
<dd><p>MAX_UTILIZATION packs as many requests as the underlying TRT engine can support in any iteration of the InflightBatching generation loop. While this is expected to maximize GPU throughput, it might require that some requests be paused and restarted depending on peak KV cache memory availability. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE">
<span id="_CPPv3N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE"></span><span id="_CPPv2N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE"></span><span class="target" id="types_8h_1aae89f79e4f8c4a402fa826e3acf68428a880f95ca896ec21f5e0a981aac04d10b"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGUARANTEED_NO_EVICT</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicy20kGUARANTEED_NO_EVICTE" title="Link to this definition"></a><br /></dt>
<dd><p>GUARANTEED_NO_EVICT uses KV cache more conservatively guaranteeing that a request, once started, will run to completion without eviction. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE">
<span id="_CPPv3N12tensorrt_llm8executor21ContextChunkingPolicyE"></span><span id="_CPPv2N12tensorrt_llm8executor21ContextChunkingPolicyE"></span><span class="target" id="types_8h_1a6d4f91b57e2ce01f804855083e22a4bf"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE">
<span id="_CPPv3N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE"></span><span id="_CPPv2N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE"></span><span class="target" id="types_8h_1a6d4f91b57e2ce01f804855083e22a4bfac5d100effe3b0eee3f5b4d48d4b5a5a9"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kFIRST_COME_FIRST_SERVED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy24kFIRST_COME_FIRST_SERVEDE" title="Link to this definition"></a><br /></dt>
<dd><p>Sequential chunking, complete the unfinished context phase first. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE"></span><span class="target" id="types_8h_1a6d4f91b57e2ce01f804855083e22a4bfac263978419a580b4ef16a199b050f89c"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kEQUAL_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicy15kEQUAL_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Iterate through each context request in sequence and attempt to increase its chunk count until the constraint is exceeded. </p>
</dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationTypeE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationTypeE"></span><span class="target" id="types_8h_1a93b2af939dbc2cc1690fb10232675ba0"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CommunicationType</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationTypeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationType4kMPIE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationType4kMPIE"></span><span class="target" id="types_8h_1a93b2af939dbc2cc1690fb10232675ba0a4352f81c41aaa6b77e87c062d4347e2b"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMPI</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationType4kMPIE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationModeE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationModeE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationModeE"></span><span class="target" id="types_8h_1ace2966b11ce94befb6438957e8aef1a4"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">CommunicationMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationModeE" title="Link to this definition"></a><br /></dt>
<dd><p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationMode7kLEADERE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationMode7kLEADERE"></span><span class="target" id="types_8h_1ace2966b11ce94befb6438957e8aef1a4a211ca40446bff2e65500e3e544664df7"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kLEADER</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode7kLEADERE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE">
<span id="_CPPv3N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE"></span><span id="_CPPv2N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE"></span><span class="target" id="types_8h_1ace2966b11ce94befb6438957e8aef1a4a88924e2c20e89e79d5782505f7228158"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kORCHESTRATOR</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor17CommunicationMode13kORCHESTRATORE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</dd></dl>
<dl class="cpp enum-class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStageE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStageE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStageE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097ca"></span><span class="k"><span class="pre">enum</span></span><span class="w"> </span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestStage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="Link to this definition"></a><br /></dt>
<dd><p>Enum class that represents the state of a request. </p>
<p><em>Values:</em></p>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage7kQUEUEDE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage7kQUEUEDE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage7kQUEUEDE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caaa2247736a3991a2eb2fe91dd90b2f6af"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kQUEUED</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage7kQUEUEDE" title="Link to this definition"></a><br /></dt>
<dd><p><a class="reference internal" href="#classtensorrt__llm_1_1executor_1_1Request"><span class="std std-ref">Request</span></a> that have been received but not yet included in the active requests (due to constraints such as maximum batch size for example). </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caaf777328746e427f4925d2423d7722757"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kENCODER_IN_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kENCODER_IN_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request in encoder phase. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caa0d34fe21e6c4aeae3dc8b16b2626ad0a"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kCONTEXT_IN_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kCONTEXT_IN_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request in context phase. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caa00590ca88f1c0cfb06a0498c299476d4"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGENERATION_IN_PROGRESS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage23kGENERATION_IN_PROGRESSE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request in generation phase. </p>
</dd></dl>
<dl class="cpp enumerator">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE"></span><span class="target" id="types_8h_1a557b39c71d92e2c17116c4972e5097caab437475b0e4a8e154f8af3ba70c51dc3"></span><span class="k"><span class="pre">enumerator</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kGENERATION_COMPLETE</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStage20kGENERATION_COMPLETEE" title="Link to this definition"></a><br /></dt>
<dd><p>Active request for which generation has completed. </p>
</dd></dl>
</dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-functions">Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy">
<span id="_CPPv3N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy"></span><span id="_CPPv2N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy"></span><span id="tensorrt_llm::executor::lshift-operator__osR.CapacitySchedulerPolicy"></span><span class="target" id="types_8h_1a263a5c1e1dba10d4f4979385a48218fa"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor23CapacitySchedulerPolicyE" title="tensorrt_llm::executor::CapacitySchedulerPolicy"><span class="n"><span class="pre">CapacitySchedulerPolicy</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">policy</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE23CapacitySchedulerPolicy" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy">
<span id="_CPPv3N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy"></span><span id="_CPPv2N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy"></span><span id="tensorrt_llm::executor::lshift-operator__osR.ContextChunkingPolicy"></span><span class="target" id="types_8h_1a0d8866f6695871181bbcfee77d5d10a2"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">&lt;&lt;</span></span></span><span class="sig-paren">(</span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">ostream</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">os</span></span>, <a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21ContextChunkingPolicyE" title="tensorrt_llm::executor::ContextChunkingPolicy"><span class="n"><span class="pre">ContextChunkingPolicy</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">policy</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executorlsERNSt7ostreamE21ContextChunkingPolicy" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0_bEN12tensorrt_llm8executor10TypeTraitsE">
<span id="_CPPv3I0_bEN12tensorrt_llm8executor10TypeTraitsE"></span><span id="_CPPv2I0_bEN12tensorrt_llm8executor10TypeTraitsE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="k"><span class="pre">false</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><a class="headerlink" href="#_CPPv4I0_bEN12tensorrt_llm8executor10TypeTraitsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>For converting a C++ data type to a <code class="docutils literal notranslate"><span class="pre">TrtLmmDataType</span></code>. </p>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIfEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsIfEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsIfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01float_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">float</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIfEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsIfE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsIfE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsIfE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:float:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01float_01_4_1aa3c4709453f3b513d302e3b2e843a464"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP32E" title="tensorrt_llm::executor::DataType::kFP32"><span class="n"><span class="pre">kFP32</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIfE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsI4halfEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsI4halfEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsI4halfEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01half_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="n"><span class="pre">half</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsI4halfEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:half:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01half_01_4_1a8b0cc854df1f430f8e8eea8f3b8a733b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kFP16E" title="tensorrt_llm::executor::DataType::kFP16"><span class="n"><span class="pre">kFP16</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsI4halfE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19PhonyNameDueToError5valueE">
<span id="_CPPv3N12tensorrt_llm8executor19PhonyNameDueToError5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor19PhonyNameDueToError5valueE"></span><span id="tensorrt_llm::executor::PhonyNameDueToError::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int8__t_01_4_1aaed76141f49476650f91cfbec3367e13"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kINT8E" title="tensorrt_llm::executor::DataType::kINT8"><span class="n"><span class="pre">kINT8</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19PhonyNameDueToError5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int32_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int32__t_01_4_1a506e37fcee8102b90f320257e12ec485"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT32E" title="tensorrt_llm::executor::DataType::kINT32"><span class="n"><span class="pre">kINT32</span></span></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">int64_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1int64__t_01_4_1ae1e174a531a0abf5c31ed44525da252d"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="tensorrt_llm::executor::DataType::kINT64"><span class="n"><span class="pre">kINT64</span></span></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIbEE">
<span id="_CPPv3IEN12tensorrt_llm8executor10TypeTraitsIbEE"></span><span id="_CPPv2IEN12tensorrt_llm8executor10TypeTraitsIbEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01bool_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><span class="kt"><span class="pre">bool</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4IEN12tensorrt_llm8executor10TypeTraitsIbEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsIbE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsIbE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsIbE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:b:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01bool_01_4_1a0cf040c47d93165f569715a3e02f22ac"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType5kBOOLE" title="tensorrt_llm::executor::DataType::kBOOL"><span class="n"><span class="pre">kBOOL</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIbE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4"></span><span class="sig-name descname"><span class="pre">template&lt;&gt;</span> <span class="pre">uint8_t</span> <span class="pre">&gt;</span></span></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp">
<span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01std_1_1uint8__t_01_4_1a36e64efaf2f35f9e9d97c62fb4e74f84"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kUINT8E" title="tensorrt_llm::executor::DataType::kUINT8"><span class="n"><span class="pre">kUINT8</span></span></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE">
<span id="_CPPv3I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE"></span><span id="_CPPv2I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE"></span><span class="k"><span class="pre">template</span></span><span class="p"><span class="pre">&lt;</span></span><span class="k"><span class="pre">typename</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">T</span></span></span><span class="p"><span class="pre">&gt;</span></span><br /><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01T_01_5_01_4"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TypeTraits</span></span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE" title="tensorrt_llm::executor::TypeTraits&lt;T*&gt;::T"><span class="n"><span class="pre">T</span></span></a><span class="p"><span class="pre">*</span></span><span class="p"><span class="pre">&gt;</span></span><a class="headerlink" href="#_CPPv4I0EN12tensorrt_llm8executor10TypeTraitsIP1TEE" title="Link to this definition"></a><br /></dt>
<dd><div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-attributes">Public Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE">
<span id="_CPPv3N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE"></span><span id="_CPPv2N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE"></span><span id="tensorrt_llm::executor::TypeTraits:TP:::value__auto"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1TypeTraits_3_01T_01_5_01_4_1a919c622815440975c53c2ba6f5e02c34"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">value</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataTypeE" title="tensorrt_llm::executor::DataType"><span class="n"><span class="pre">DataType</span></span></a><span class="p"><span class="pre">::</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor8DataType6kINT64E" title="tensorrt_llm::executor::DataType::kINT64"><span class="n"><span class="pre">kINT64</span></span></a><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor10TypeTraitsIP1TE5valueE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStatsE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStatsE"></span><span id="tensorrt_llm::executor::KvCacheStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">KvCacheStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a KV cache manager. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::maxNumBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a2952c746c5724a096a0f90037cc95a9b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12maxNumBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Max number of blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::freeNumBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a4393a6d986ba543b215c761888732ddc"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">freeNumBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13freeNumBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of free blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::usedNumBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a9cd563d7a3d9a49a7bea72fbb386fd0c"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usedNumBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats13usedNumBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of used blocks. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE"></span><span id="tensorrt_llm::executor::KvCacheStats::tokensPerBlock__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a7b0c51dae64cd0af386fb0f0104dedc8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">tokensPerBlock</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14tokensPerBlockE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of tokens per block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::allocTotalBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a88f44cd389cb3813a93245cab0aa96d8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocTotalBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats16allocTotalBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of total allocated block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::allocNewBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a804a58fa6ad301b7ddfe922cb413d265"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allocNewBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats14allocNewBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of newly allocated block. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE">
<span id="_CPPv3N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE"></span><span id="_CPPv2N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE"></span><span id="tensorrt_llm::executor::KvCacheStats::reusedBlocks__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1KvCacheStats_1a1c54f95634272634dcb7779bfd87e162"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">reusedBlocks</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStats12reusedBlocksE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of reused block. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStatsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">StaticBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of static batching models for a single iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numScheduledRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a85a29e377cc0afc1fa3b5ac5e2426509"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numScheduledRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats20numScheduledRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of scheduled requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numContextRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a44b1d7847625f6042de40ef2a5c8ec70"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numContextRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats18numContextRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests in context stage. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numCtxTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1aed7aa230c825577f4acb9d43a6f5176d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCtxTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numCtxTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of context tokens in the iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::numGenTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a48522c73455172970cb9dc292e53fcd3"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGenTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats12numGenTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of tokens to generate in the iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE">
<span id="_CPPv3N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE"></span><span id="_CPPv2N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE"></span><span id="tensorrt_llm::executor::StaticBatchingStats::emptyGenSlots__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1StaticBatchingStats_1a3316815cfbe07bb2a486de89b7024c41"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">emptyGenSlots</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStats13emptyGenSlotsE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of unused generation token slots. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStatsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">InflightBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of inflight batching models for a single iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numScheduledRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1adf6843f2538709caad5542216b13693e"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numScheduledRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats20numScheduledRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of scheduled requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numContextRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1afc34637bdddb02cb101f38a9c8a4a50a"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numContextRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats18numContextRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests in context stage. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numGenRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a226f8ec68000216bfeedc6040e08da7d"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGenRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats14numGenRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of requests in generation stage. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numPausedRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a942c50f8eaba7dd00ded9f10e257f286"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numPausedRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats17numPausedRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of paused requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::numCtxTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a052d62534092ccc9824332a1d55da3a8"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numCtxTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12numCtxTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>Total number of context tokens in the iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::microBatchId__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1a945be330caa27de5a173aec6d71a6b03"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">microBatchId</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats12microBatchIdE" title="Link to this definition"></a><br /></dt>
<dd><p>Index of mirco batch. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE">
<span id="_CPPv3N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE"></span><span id="_CPPv2N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE"></span><span id="tensorrt_llm::executor::InflightBatchingStats::avgNumDecodedTokensPerIter__float"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1InflightBatchingStats_1acca77d655808fe504c6f99c8c6e9c456"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">avgNumDecodedTokensPerIter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStats26avgNumDecodedTokensPerIterE" title="Link to this definition"></a><br /></dt>
<dd><p>Average number of tokens decoded per request per iteration. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStatsE"></span><span id="tensorrt_llm::executor::IterationStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">IterationStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a single iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats9timestampE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats9timestampE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats9timestampE"></span><span id="tensorrt_llm::executor::IterationStats::timestamp__ss"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1af1d7e6fb431b3bae3be69d0cc3a6b00e"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">string</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">timestamp</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats9timestampE" title="Link to this definition"></a><br /></dt>
<dd><p>Ending time of this iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats4iterE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats4iterE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats4iterE"></span><span id="tensorrt_llm::executor::IterationStats::iter__IterationType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ae19cff294da29d71592cabdc54be9774"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats4iterE" title="Link to this definition"></a><br /></dt>
<dd><p>Iteration id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats13iterLatencyMSE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats13iterLatencyMSE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats13iterLatencyMSE"></span><span id="tensorrt_llm::executor::IterationStats::iterLatencyMS__double"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a08d812da19f047a777e11e217b82eaa8"></span><span class="kt"><span class="pre">double</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iterLatencyMS</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats13iterLatencyMSE" title="Link to this definition"></a><br /></dt>
<dd><p>Iteration latency (ms) </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats17numActiveRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats17numActiveRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats17numActiveRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::numActiveRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a7b8b4c8acde99a1eb3de70050e770458"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numActiveRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numActiveRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of active requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::numQueuedRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1abddd1933dca91cbf6336b10e3fdf6e1b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numQueuedRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17numQueuedRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of queued requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE"></span><span id="tensorrt_llm::executor::IterationStats::maxNumActiveRequests__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac8c5c8bf27f8c0eec9fa7ae41da3fe67"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">maxNumActiveRequests</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats20maxNumActiveRequestsE" title="Link to this definition"></a><br /></dt>
<dd><p>Number of max active requests. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats11gpuMemUsageE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats11gpuMemUsageE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats11gpuMemUsageE"></span><span id="tensorrt_llm::executor::IterationStats::gpuMemUsage__s"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac975025c2e40a167051e4c79afcea908"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">gpuMemUsage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11gpuMemUsageE" title="Link to this definition"></a><br /></dt>
<dd><p>GPU memory usage in bytes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats11cpuMemUsageE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats11cpuMemUsageE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats11cpuMemUsageE"></span><span id="tensorrt_llm::executor::IterationStats::cpuMemUsage__s"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1abba95b563f5fc8459b626b05472eef38"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">cpuMemUsage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats11cpuMemUsageE" title="Link to this definition"></a><br /></dt>
<dd><p>CPU memory usage in bytes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE"></span><span id="tensorrt_llm::executor::IterationStats::pinnedMemUsage__s"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a99f299d90d3366ae1f694890fd859455"></span><span class="n"><span class="pre">size_t</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">pinnedMemUsage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats14pinnedMemUsageE" title="Link to this definition"></a><br /></dt>
<dd><p>Pinned memory usage in bytes. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats12kvCacheStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats12kvCacheStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats12kvCacheStatsE"></span><span id="tensorrt_llm::executor::IterationStats::kvCacheStats__std::optional:KvCacheStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1ac234045a5fdd46880205b99f6d326977"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kvCacheStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats12kvCacheStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to KV caches. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE"></span><span id="tensorrt_llm::executor::IterationStats::crossKvCacheStats__std::optional:KvCacheStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a3f985a83e7ba30a463c13c4cafff87fe"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12KvCacheStatsE" title="tensorrt_llm::executor::KvCacheStats"><span class="n"><span class="pre">KvCacheStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">crossKvCacheStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats17crossKvCacheStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to cross KV caches. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE"></span><span id="tensorrt_llm::executor::IterationStats::staticBatchingStats__std::optional:StaticBatchingStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a361f059648067ed27ccd1baa91656e8f"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor19StaticBatchingStatsE" title="tensorrt_llm::executor::StaticBatchingStats"><span class="n"><span class="pre">StaticBatchingStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">staticBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats19staticBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to static batching. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE">
<span id="_CPPv3N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE"></span><span id="tensorrt_llm::executor::IterationStats::inflightBatchingStats__std::optional:InflightBatchingStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1IterationStats_1a8a9f5f9365d9c7cb7e2adcb5e76fb811"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">optional</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor21InflightBatchingStatsE" title="tensorrt_llm::executor::InflightBatchingStats"><span class="n"><span class="pre">InflightBatchingStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">inflightBatchingStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor14IterationStats21inflightBatchingStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>Stats specific to inflight batching. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStatsE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStatsE"></span><span id="tensorrt_llm::executor::RequestStats"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of a single request. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats2idE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats2idE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats2idE"></span><span id="tensorrt_llm::executor::RequestStats::id__IdType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ae069f9ba9675130e5c3e3b9129d2aff1"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor6IdTypeE" title="tensorrt_llm::executor::IdType"><span class="n"><span class="pre">IdType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">id</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats2idE" title="Link to this definition"></a><br /></dt>
<dd><p>The request id. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats5stageE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats5stageE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats5stageE"></span><span id="tensorrt_llm::executor::RequestStats::stage__RequestStage"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a6d35e8a95e6cccaceae38961edd5a354"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStageE" title="tensorrt_llm::executor::RequestStage"><span class="n"><span class="pre">RequestStage</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">stage</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats5stageE" title="Link to this definition"></a><br /></dt>
<dd><p>The current stage the request is in. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE"></span><span id="tensorrt_llm::executor::RequestStats::contextPrefillPosition__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ae2372e9fb63311f9f95f77cdf8d09f5b"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">contextPrefillPosition</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats22contextPrefillPositionE" title="Link to this definition"></a><br /></dt>
<dd><p>If using chunked context, the current context prefill position. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE"></span><span id="tensorrt_llm::executor::RequestStats::numGeneratedTokens__SizeType32"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a18f2badb3c0c81ffee60f7f8e22d2b99"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">numGeneratedTokens</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats18numGeneratedTokensE" title="Link to this definition"></a><br /></dt>
<dd><p>The number of generated tokens so far. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE"></span><span id="tensorrt_llm::executor::RequestStats::avgNumDecodedTokensPerIter__float"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a44573f7d4ab8fce4a931c661929deab1"></span><span class="kt"><span class="pre">float</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">avgNumDecodedTokensPerIter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats26avgNumDecodedTokensPerIterE" title="Link to this definition"></a><br /></dt>
<dd><p>The average number of decoded tokens per iteration. It is &gt;= 1 for speculative decoding. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats9scheduledE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats9scheduledE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats9scheduledE"></span><span id="tensorrt_llm::executor::RequestStats::scheduled__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1ab928aa93fcf7a22e40adfd10a62aee04"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">scheduled</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats9scheduledE" title="Link to this definition"></a><br /></dt>
<dd><p>Whether the request is scheduled for the current iteration. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12RequestStats6pausedE">
<span id="_CPPv3N12tensorrt_llm8executor12RequestStats6pausedE"></span><span id="_CPPv2N12tensorrt_llm8executor12RequestStats6pausedE"></span><span id="tensorrt_llm::executor::RequestStats::paused__b"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStats_1a6f74bde53fc8910bad82297f8f741deb"></span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">paused</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12RequestStats6pausedE" title="Link to this definition"></a><br /></dt>
<dd><p>Whether the request is being paused at the current iteration due to lack of resources (KV cache blocks exhaustion for example) </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp struct">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE">
<span id="_CPPv3N12tensorrt_llm8executor24RequestStatsPerIterationE"></span><span id="_CPPv2N12tensorrt_llm8executor24RequestStatsPerIterationE"></span><span id="tensorrt_llm::executor::RequestStatsPerIteration"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStatsPerIteration"></span><span class="k"><span class="pre">struct</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">RequestStatsPerIteration</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIterationE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>Struct that holds the stats of all requests in an iteration. </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-members">Public Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration4iterE">
<span id="_CPPv3N12tensorrt_llm8executor24RequestStatsPerIteration4iterE"></span><span id="_CPPv2N12tensorrt_llm8executor24RequestStatsPerIteration4iterE"></span><span id="tensorrt_llm::executor::RequestStatsPerIteration::iter__IterationType"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStatsPerIteration_1a089fa8d28441ffb3d0f37bd046329ac7"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor13IterationTypeE" title="tensorrt_llm::executor::IterationType"><span class="n"><span class="pre">IterationType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">iter</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration4iterE" title="Link to this definition"></a><br /></dt>
<dd><p>The iteration id for these stats. </p>
</dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE">
<span id="_CPPv3N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE"></span><span id="_CPPv2N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE"></span><span id="tensorrt_llm::executor::RequestStatsPerIteration::requestStats__std::vector:RequestStats:"></span><span class="target" id="structtensorrt__llm_1_1executor_1_1RequestStatsPerIteration_1af9fbfa79e3985aa6f824b1d7edf934d0"></span><span class="n"><span class="pre">std</span></span><span class="p"><span class="pre">::</span></span><span class="n"><span class="pre">vector</span></span><span class="p"><span class="pre">&lt;</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12RequestStatsE" title="tensorrt_llm::executor::RequestStats"><span class="n"><span class="pre">RequestStats</span></span></a><span class="p"><span class="pre">&gt;</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">requestStats</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor24RequestStatsPerIteration12requestStatsE" title="Link to this definition"></a><br /></dt>
<dd><p>The stats of all active requests for this iteration. </p>
</dd></dl>
</div>
</dd></dl>
<dl class="cpp class">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingModeE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingModeE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingModeE"></span><span id="tensorrt_llm::executor::DecodingMode"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode"></span><span class="k"><span class="pre">class</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingMode</span></span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="Link to this definition"></a><br /></dt>
<dd><div class="docutils container">
<em>#include &lt;types.h&gt;</em></div>
<p>mode of the decoder </p>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-types">Public Types</p>
<dl class="cpp type">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a3448ce5ea22fc30852625ab0fef44d7c"></span><span class="k"><span class="pre">using</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">UnderlyingType</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="n"><span class="pre">uint32_t</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-functions">Public Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode14useTemperatureEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode14useTemperatureEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode14useTemperatureEb"></span><span id="tensorrt_llm::executor::DecodingMode::useTemperature__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a2210c295976ce01a6f8d9552316fa57c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useTemperature</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useTemp</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14useTemperatureEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb"></span><span id="tensorrt_llm::executor::DecodingMode::useOccurrencePenalties__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a447bc29535c01a0a3fabb696a770a6e1"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useOccurrencePenalties</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22useOccurrencePenaltiesEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb"></span><span id="tensorrt_llm::executor::DecodingMode::usePresencePenalty__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a824006c164e14ecf88668123ffad3147"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">usePresencePenalty</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18usePresencePenaltyEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb"></span><span id="tensorrt_llm::executor::DecodingMode::useRepetitionPenalty__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a092514e0af8fac76618628bbcb8c4de8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useRepetitionPenaltyEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb"></span><span id="tensorrt_llm::executor::DecodingMode::useFrequencyPenalty__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a8f874092e9748988972c4cdb57bb774c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">usePenalty</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19useFrequencyPenaltyEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12useMinLengthEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12useMinLengthEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12useMinLengthEb"></span><span id="tensorrt_llm::executor::DecodingMode::useMinLength__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a2631da66edab39cf70f1c9b1ff5a7e1d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useMinLength</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">useMinLen</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useMinLengthEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12useBanTokensEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12useBanTokensEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12useBanTokensEb"></span><span id="tensorrt_llm::executor::DecodingMode::useBanTokens__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ac6fb45f8e99b111d075398e5ab8a0f2a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useBanTokens</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">banTokens</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useBanTokensEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode11useBanWordsEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode11useBanWordsEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode11useBanWordsEb"></span><span id="tensorrt_llm::executor::DecodingMode::useBanWords__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a00c2f297b1e339c1a35e495a9736e282"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useBanWords</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">banWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11useBanWordsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb"></span><span id="tensorrt_llm::executor::DecodingMode::useNoRepeatNgramSize__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1343a44b16892f9dc6502e51daf9d88c"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">noRepeatNgramSize</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20useNoRepeatNgramSizeEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12useStopWordsEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12useStopWordsEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12useStopWordsEb"></span><span id="tensorrt_llm::executor::DecodingMode::useStopWords__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1afd5a7f53e18bd79a5447e12f255fb4f8"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useStopWords</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">stopWords</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12useStopWordsEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb"></span><span id="tensorrt_llm::executor::DecodingMode::useMaxLengthStop__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a6fdffddebb1e8a75bcadfc497853b5da"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useMaxLengthStop</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">maxLengthStop</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode16useMaxLengthStopEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb"></span><span id="tensorrt_llm::executor::DecodingMode::useExplicitEosStop__bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af99c1d9592b789355c5dae453a6b1b1d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">useExplicitEosStop</span></span></span><span class="sig-paren">(</span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">explicitEosStop</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode18useExplicitEosStopEb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode6isAutoEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode6isAutoEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode6isAutoEv"></span><span id="tensorrt_llm::executor::DecodingMode::isAutoCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a03fca2f112cf20696b5c519bce1d7d01"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isAuto</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isAutoEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopKEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode6isTopKEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode6isTopKEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopKCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aed23fc36d8e39b2b0b58bcca8f2d656d"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopK</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopKEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode6isTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode6isTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopPCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ada1c8497d1327eb7ad1b19256cb86549"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode6isTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopKorTopPCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af06c17d97d5b07c7a2b9a929e94227dd"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopKorTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isTopKorTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::isTopKandTopPCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aa6de87202e3216f15e138238dd913196"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isTopKandTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isTopKandTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv"></span><span id="tensorrt_llm::executor::DecodingMode::isBeamSearchCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a98fd1b390d25b2f721f4f29fdadd696b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isBeamSearch</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isBeamSearchEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode8isMedusaEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode8isMedusaEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode8isMedusaEv"></span><span id="tensorrt_llm::executor::DecodingMode::isMedusaCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ad79679f0f97bce7c6aba223393acb6fb"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isMedusa</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8isMedusaEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv"></span><span id="tensorrt_llm::executor::DecodingMode::isLookaheadCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a4e38cc4354a5037195f3b8a1c087aa0e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isLookahead</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode11isLookaheadEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::isExplicitDraftTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1abeb845437eb6024512f6048d3759b15f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isExplicitDraftTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isExplicitDraftTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseTemperatureCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1afc7155f69db8ab7a85f1e50eee532239"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseTemperature</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode16isUseTemperatureEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUsePresencePenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ae37387b8b17ec067d93b336e380ced8e"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUsePresencePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUsePresencePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseFrequencyPenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a8771a4cedc4ece67e6aa29ae3585c231"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseFrequencyPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode21isUseFrequencyPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseRepetitionPenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a2de0d1d2a1b4c830d9598fca97b6d180"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseRepetitionPenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseRepetitionPenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseMinLengthCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a6523a66ebcf6780297ffc18b6cc11a51"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseMinLength</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseMinLengthEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseOccurrencePenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a92eeaf8443658d34303ee04ed7acccc0"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseOccurrencePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseOccurrencePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUsePenaltyCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a261919999fa5c9f7941003c827ea37ed"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUsePenalty</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode12isUsePenaltyEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseBanWordsCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aa4abd8e3a477811ea531f1a91a8a4aad"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseBanWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode13isUseBanWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseNoRepeatNgramSizeCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a20611dff4d94683117ff8ec8c8691f3f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseNoRepeatNgramSize</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode22isUseNoRepeatNgramSizeEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseBanTokensCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1adce098d4a048c94633490fb0a96e4827"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseBanTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseBanTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseStopWordsCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a80652680f829fa04d3436ed35891e8ab"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseStopWords</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode14isUseStopWordsEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseMaxLengthStopCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a4d2c6bc1638c61cc897d86232b425479"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseMaxLengthStop</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode18isUseMaxLengthStopEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseExplicitEosStopCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aaee0761190422e82d7a5c6a8358f95f2"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseExplicitEosStop</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode20isUseExplicitEosStopEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv"></span><span id="tensorrt_llm::executor::DecodingMode::isUseStopCriteriaCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a3e2e72d562de18021261b95d593ea231"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">isUseStopCriteria</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode17isUseStopCriteriaEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode"></span><span id="tensorrt_llm::executor::DecodingMode::eq-operator__DecodingModeCRC"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aaa854d9caaad142db1f8f0cbfcba8f8f"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="k"><span class="pre">operator</span></span><span class="o"><span class="pre">==</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingModeE" title="tensorrt_llm::executor::DecodingMode"><span class="n"><span class="pre">DecodingMode</span></span></a><span class="w"> </span><span class="k"><span class="pre">const</span></span><span class="w"> </span><span class="p"><span class="pre">&amp;</span></span><span class="n sig-param"><span class="pre">other</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingModeeqERK12DecodingMode" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType"></span><span id="tensorrt_llm::executor::DecodingMode::DecodingMode__UnderlyingTypeCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a844ac72db3cf0d3961dc9a90f86e101b"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">explicit</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">DecodingMode</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">state</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12DecodingModeE14UnderlyingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode8getStateEv">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode8getStateEv"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode8getStateEv"></span><span id="tensorrt_llm::executor::DecodingMode::getStateCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a27d6ca8f5f71b63b26dc7eb2b3d03cfc"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">getState</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode8getStateEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-public-static-functions">Public Static Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode4AutoEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode4AutoEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode4AutoEv"></span><span id="tensorrt_llm::executor::DecodingMode::AutoCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a6c339009e02105fba031e0c9710a3790"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Auto</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4AutoEv" title="Link to this definition"></a><br /></dt>
<dd><p>No mode specified. Config will be determined from the beam width of the first request at runtime TopKTopP if beamWidth == 1, BeamSearch otherwise. </p>
</dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode4TopKEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode4TopKEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode4TopKEv"></span><span id="tensorrt_llm::executor::DecodingMode::TopKCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ad9bf14290ab3d9a89a9ee2c38682b504"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TopK</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopKEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode4TopPEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode4TopPEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode4TopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::TopPCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a09a2f30b9ce6a5fca2df259e90a6b355"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode4TopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode8TopKTopPEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode8TopKTopPEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode8TopKTopPEv"></span><span id="tensorrt_llm::executor::DecodingMode::TopKTopPCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a902c590ed087e57afd640fcdf3a843c5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">TopKTopP</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8TopKTopPEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode10BeamSearchEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode10BeamSearchEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode10BeamSearchEv"></span><span id="tensorrt_llm::executor::DecodingMode::BeamSearchCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af623b5711fccf09e701a80ac9040d056"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">BeamSearch</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10BeamSearchEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode6MedusaEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode6MedusaEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode6MedusaEv"></span><span id="tensorrt_llm::executor::DecodingMode::MedusaCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a0c844cc3460c5c49e235d41fc8ee86ae"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Medusa</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6MedusaEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode9LookaheadEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode9LookaheadEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode9LookaheadEv"></span><span id="tensorrt_llm::executor::DecodingMode::LookaheadCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a56c82a00093864ac7b66fd4b7b187879"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">Lookahead</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9LookaheadEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv"></span><span id="tensorrt_llm::executor::DecodingMode::ExplicitDraftTokensCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a73347d25259681d45a916b64daef7ce7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">auto</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">ExplicitDraftTokens</span></span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19ExplicitDraftTokensEv" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-functions">Private Functions</p>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType"></span><span id="tensorrt_llm::executor::DecodingMode::anyBitSet__UnderlyingTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ad55eaeac9dbf85d0e99cd619150e5881"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">anyBitSet</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9anyBitSetE14UnderlyingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType">
<span id="_CPPv3NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType"></span><span id="_CPPv2NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType"></span><span id="tensorrt_llm::executor::DecodingMode::allBitSet__UnderlyingTypeCCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ae5c2de19684a00fcf41ecd4c750177df"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">allBitSet</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">bits</span></span><span class="sig-paren">)</span><span class="w"> </span><span class="k"><span class="pre">const</span></span><a class="headerlink" href="#_CPPv4NK12tensorrt_llm8executor12DecodingMode9allBitSetE14UnderlyingType" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp function">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb"></span><span id="tensorrt_llm::executor::DecodingMode::setBitTo__UnderlyingType.bCE"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a158da147bd5b22de369c67e33ba4ab9a"></span><span class="k"><span class="pre">inline</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">setBitTo</span></span></span><span class="sig-paren">(</span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="n sig-param"><span class="pre">state</span></span>, <span class="kt"><span class="pre">bool</span></span><span class="w"> </span><span class="n sig-param"><span class="pre">x</span></span><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode8setBitToE14UnderlyingTypeb" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-members">Private Members</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode6mStateE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode6mStateE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode6mStateE"></span><span id="tensorrt_llm::executor::DecodingMode::mState__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a7d858720e8c48b761bfb6fae613cd69f"></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">mState</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode6mStateE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
<div class="breathe-sectiondef docutils container">
<p class="breathe-sectiondef-title rubric" id="breathe-section-title-private-static-attributes">Private Static Attributes</p>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseRepetitionPenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1afe5e324276fcdea07454c19f6c0e37bf"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseRepetitionPenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseFrequencyPenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ad85aa4ad66cb9dc481207e39264c982a"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseFrequencyPenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUsePresencePenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af7a47b63526a2cba1bd6116518f066cd"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUsePresencePenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseTemperature__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1ad022f99181a56c6ee3638f4ea288da55"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseTemperature</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">3</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseMinLength__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a56874b0868fc3c1a415a38c985c9d9ce"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseMinLength</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">4</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseBanWords__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aeeb3477230f6b10da738c2464f182900"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseBanWords</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">5</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseStopWords__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a7ca21e04b82545ea3fd00b785549ab1c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseStopWords</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">6</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseMaxLengthStop__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a42274e9eec90521453b71c49df6814b4"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseMaxLengthStop</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">7</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseExplicitEosStop__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af3b4ccdc681f7d5d6ea644dedd27f6a5"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseExplicitEosStop</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">8</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode19kUseExplicitEosStopE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseNoRepeatNgramSize__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a2ff7f9600fd679af27a5330d046e40be"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseNoRepeatNgramSize</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="m"><span class="pre">9</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE"></span><span id="tensorrt_llm::executor::DecodingMode::kStandardStopCriteria__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a14b9a1bb7c4236b311000282665be757"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kStandardStopCriteria</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseStopWordsE" title="tensorrt_llm::executor::DecodingMode::kUseStopWords"><span class="n"><span class="pre">kUseStopWords</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode17kUseMaxLengthStopE" title="tensorrt_llm::executor::DecodingMode::kUseMaxLengthStop"><span class="n"><span class="pre">kUseMaxLengthStop</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kStandardStopCriteriaE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseOccurrencePenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a27d247f55e66e4d0b63806071577f488"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseOccurrencePenalties</span></span></span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseRepetitionPenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUseRepetitionPenalties"><span class="n"><span class="pre">kUseRepetitionPenalties</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode22kUseFrequencyPenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUseFrequencyPenalties"><span class="n"><span class="pre">kUseFrequencyPenalties</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUsePresencePenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUsePresencePenalties"><span class="n"><span class="pre">kUsePresencePenalties</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE"></span><span id="tensorrt_llm::executor::DecodingMode::kUsePenalties__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aa29037dc381264b5f249c8459ad8d5c7"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUsePenalties</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode23kUseOccurrencePenaltiesE" title="tensorrt_llm::executor::DecodingMode::kUseOccurrencePenalties"><span class="n"><span class="pre">kUseOccurrencePenalties</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode15kUseTemperatureE" title="tensorrt_llm::executor::DecodingMode::kUseTemperature"><span class="n"><span class="pre">kUseTemperature</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseMinLengthE" title="tensorrt_llm::executor::DecodingMode::kUseMinLength"><span class="n"><span class="pre">kUseMinLength</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUsePenaltiesE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE"></span><span id="tensorrt_llm::executor::DecodingMode::kUseBanTokens__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a8a8838699785e2c5ca13d607ad19db07"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kUseBanTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode21kUseNoRepeatNgramSizeE" title="tensorrt_llm::executor::DecodingMode::kUseNoRepeatNgramSize"><span class="n"><span class="pre">kUseNoRepeatNgramSize</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode12kUseBanWordsE" title="tensorrt_llm::executor::DecodingMode::kUseBanWords"><span class="n"><span class="pre">kUseBanWords</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode13kUseBanTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode9kNumFlagsE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode9kNumFlagsE"></span><span id="tensorrt_llm::executor::DecodingMode::kNumFlags__SizeType32"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a1d2c67591279f28877bff71b117ad83b"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor10SizeType32E" title="tensorrt_llm::executor::SizeType32"><span class="n"><span class="pre">SizeType32</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kNumFlags</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">10</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5kAutoE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5kAutoE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5kAutoE"></span><span id="tensorrt_llm::executor::DecodingMode::kAuto__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a7c76d7300182eba3d6f69ea2333f796c"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kAuto</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">0</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kAutoE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5kTopKE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5kTopKE"></span><span id="tensorrt_llm::executor::DecodingMode::kTopK__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1af958baee817839c7b889755c753f46e6"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTopK</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">1</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode5kTopPE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode5kTopPE"></span><span id="tensorrt_llm::executor::DecodingMode::kTopP__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a5ced226c5010159b92c7a1e50c26ed4f"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTopP</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">2</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode11kBeamSearchE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode11kBeamSearchE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode11kBeamSearchE"></span><span id="tensorrt_llm::executor::DecodingMode::kBeamSearch__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1aadcd352bbf891386b85f7d9b46b82288"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kBeamSearch</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">3</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode11kBeamSearchE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode7kMedusaE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode7kMedusaE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode7kMedusaE"></span><span id="tensorrt_llm::executor::DecodingMode::kMedusa__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a908ff7a30c2df93f0fabef415fafeb49"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kMedusa</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">4</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode7kMedusaE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode10kLookaheadE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode10kLookaheadE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode10kLookaheadE"></span><span id="tensorrt_llm::executor::DecodingMode::kLookahead__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a938ba770844ccc1cdf20cca1876c8d40"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kLookahead</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">5</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode10kLookaheadE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE"></span><span id="tensorrt_llm::executor::DecodingMode::kExplicitDraftTokens__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a64ef065273ebd6bd441eb56a69cada44"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kExplicitDraftTokens</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><span class="m"><span class="pre">1u</span></span><span class="w"> </span><span class="o"><span class="pre">&lt;&lt;</span></span><span class="w"> </span><span class="p"><span class="pre">(</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kNumFlagsE" title="tensorrt_llm::executor::DecodingMode::kNumFlags"><span class="n"><span class="pre">kNumFlags</span></span></a><span class="w"> </span><span class="o"><span class="pre">+</span></span><span class="w"> </span><span class="m"><span class="pre">6</span></span><span class="p"><span class="pre">)</span></span><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode20kExplicitDraftTokensE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
<dl class="cpp var">
<dt class="sig sig-object cpp" id="_CPPv4N12tensorrt_llm8executor12DecodingMode9kTopKTopPE">
<span id="_CPPv3N12tensorrt_llm8executor12DecodingMode9kTopKTopPE"></span><span id="_CPPv2N12tensorrt_llm8executor12DecodingMode9kTopKTopPE"></span><span id="tensorrt_llm::executor::DecodingMode::kTopKTopP__UnderlyingType"></span><span class="target" id="classtensorrt__llm_1_1executor_1_1DecodingMode_1a008b9b50f31c83a023b1f6945f23e2d1"></span><span class="k"><span class="pre">static</span></span><span class="w"> </span><span class="k"><span class="pre">constexpr</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode14UnderlyingTypeE" title="tensorrt_llm::executor::DecodingMode::UnderlyingType"><span class="n"><span class="pre">UnderlyingType</span></span></a><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">kTopKTopP</span></span></span><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="p"><span class="pre">{</span></span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopKE" title="tensorrt_llm::executor::DecodingMode::kTopK"><span class="n"><span class="pre">kTopK</span></span></a><span class="w"> </span><span class="o"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode5kTopPE" title="tensorrt_llm::executor::DecodingMode::kTopP"><span class="n"><span class="pre">kTopP</span></span></a><span class="p"><span class="pre">}</span></span><a class="headerlink" href="#_CPPv4N12tensorrt_llm8executor12DecodingMode9kTopKTopPE" title="Link to this definition"></a><br /></dt>
<dd></dd></dl>
</div>
</dd></dl>
</dd></dl>
<dl class="cpp type">
<dt class="sig sig-object cpp">
<span class="target" id="namespacetensorrt__llm_1_1runtime"></span><span class="k"><span class="pre">namespace</span></span><span class="w"> </span><span class="sig-name descname"><span class="n"><span class="pre">runtime</span></span></span><br /></dt>
<dd></dd></dl>
</dd></dl>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="../reference/memory.html" class="btn btn-neutral float-left" title="Memory Usage of TensorRT-LLM" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="runtime.html" class="btn btn-neutral float-right" title="Runtime" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<jinja2.runtime.BlockReference object at 0x7f8e37d07880>
<div class="footer">
<p>
Copyright © 2024 NVIDIA Corporation
</p>
<p>
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/" target="_blank" rel="noopener"
data-cms-ai="0">Privacy Policy</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-center/" target="_blank" rel="noopener"
data-cms-ai="0">Manage My Privacy</a> |
<a class="Link" href="https://www.nvidia.com/en-us/preferences/start/" target="_blank" rel="noopener"
data-cms-ai="0">Do Not Sell or Share My Data</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/terms-of-service/" target="_blank"
rel="noopener" data-cms-ai="0">Terms of Service</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/accessibility/" target="_blank" rel="noopener"
data-cms-ai="0">Accessibility</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/company-policies/" target="_blank"
rel="noopener" data-cms-ai="0">Corporate Policies</a> |
<a class="Link" href="https://www.nvidia.com/en-us/product-security/" target="_blank" rel="noopener"
data-cms-ai="0">Product Security</a> |
<a class="Link" href="https://www.nvidia.com/en-us/contact/" target="_blank" rel="noopener"
data-cms-ai="0">Contact</a>
</p>
</div>
</div>
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink